Python MongoDB

Python SQLite

Python CGI Programming

In the Python CGI programming, we will learn how we can run the Python Script on the web; we will learn how Python file can be executed as CGI Script and discuss the its configuration of web-browser to Python script run as a CGI. Also, we will learn the following topics:

What is CGI?
Web Browsing
Configure Apache Web server for CGI
Using the cgi Module
CGI Architecture Diagram
Running First Python Program as CGI
Legends and syntaxes
Debugging of Python CGI Script
Functions of CGI
Advantages of CGI
Disadvantages of CGI
Common Problems and Solutions

What is CGI?

The word CGI is the acronyms of the "Common Gateway Interface", which is used to define how to exchange information between the web server and a custom script. The NCSA officially manages the CGI scripts.

The Common Gateway Interface is a standard for external gateway programs to interface with the server, such as HTTP Servers.

In simple words, it is the collection of methods used to set up a dynamic interaction between the server, and the client application. When a client sends a request to the webserver, the CGI programs execute that particular request and send back the result to the webserver.

The users may submit the information in web browser by using HTML <form> or <isindex> element. There is a server's special directory called cgi-bin, where cgi script is generally stored. When a client makes a request, the server adds the additional information to request.

This additional information can be the hostname of the client, the query string, the requested URL, and so on. The webserver executes and sends the output to the web browser (or other client application).

Sometimes the server passes the data by a query string that is the part of the URL. A query string may not be conventionally a hierarchical path structure such as the following link.

https://www.google.com/search?q=wikipedia&oq=wiki&aqs=chrome.1.69i57j0l6j5.3046j0j7&sourceid=chrome&ie=UTF-8

Python provides the CGI module, which helps to debug the script and also support for the uploading files through a HTML form.

So here the question arises what does a Python CGI script output look like? The HTTPs server gives back the output as two sections separated by a blank line. The first section grasps the number of headers, notifying the client what kind of data is following.

Let's understand the following example of generate the minimal header section in the Python CGI programming.

Example -

# HTML is following
print("Content-Type: text/html")   
# blank line, end of headers
print() 

In the above example, the first statement says that, the server that html code follows; the second blank line indicates the header is ended here. Let's understand the following example of generating the minimal header section in the Python CGI programming.

Example -

print("<title> This is a CGI script output</title>")
print("<h1>This is my first CGI script</h1>")
print("Hello, JavaTpoint!")

Web Browsing

Before understanding the CGI concepts, we should know the internal process of web page or URL when we click on the given link.

The client (web browser) communicates with the HTTP server and asks for the URL i.e., filename.
If the web browser finds that requested file, then it sends back to the client (web browser), otherwise it sends the error message to the client as error file.
Here the responsibility of the web browser to display either the received file or an error message.

However, we can set a HTTP server so that whenever user requests in a particular dictionary, then it should be sent to the client; instead, it executed as a program and whatever the result is sent back for the client to display. This process is called the Common Gateway Interface or CGI and the programs are called CGI scripts. We can write the CGI programs as Python Script, PERL, Script, Shell Script, C or C++, programs, etc.

Configure Apache Web server for CGI

We need to configure the Apache Web server in order to run the CGI script on the server.

CGI Architecture

Using the cgi module

Python provides the cgi module, which consists of many useful built-in functions. We can use them by importing the cgi module.

Now, we can write further script.

import cgi
cgitb.enable()

The above script will stimulate an exception handler to show the detailed report in the web browser of occurred errors. We can also save the report by using the following script.

import cgitb
cgitb.enable(display=0, logdir="/path/to/logdir")

The above feature of the cgi module is helpful during the script development. These reports help us to debug the script effectively. When we get the expected output, we can remove this.

Previously, we have discussed the users save information using the form. So how can we get that information? Python provides the FieldStorage class. We can apply the encoding keyword parameter to the document if the form contains the non-ASCII character. We will find the content <META> tag in the <HEAD> section in our HTML document.

The FieldStorage class reads the form information from the standard input or the environment.

A FieldStorage instance is the same as the Python dictionary. We can use the len() and all dictionary function in the FieldStorage instance. It overlooks fields with empty string values. We can also consider the empty values using the optional keyword parameter keep_blank_values by setting True.

Example -

form = cgi.FieldStorage()
if ("name" not in form or "addr" not in form):
    print("<H1>Error</H1>")
    print("Please enter the information in the name and address fields.")
    return
print("<p>name:", form["name"].value)
print("<p>addr:", form["addr"].value)
#Next lines of code will execute here...

In the above example, we have used the form ["name"], here name is key. This is used to extract the value which is entered by the user.

We can use the getvalue() method to fetch the string value directly. This function also takes an optional second argument as a default. If the key is not present, it returns the default value.

If the submitted form data have more than one field with the same name, we should use the form.getlist() function. It returns the list of strings. Look at the following code, we add the any number of username field, which is separated by commas.

value1 = form.getlist("username")
usernames1 = ",".join(value)

If the field is uploaded file, then it can be accessed by value attribute or the getvalue() method and read that uploaded file in bytes. Let's understand the following code if user upload the file.

Example -

file_item = form["userfile"]
if (fileitem.file):
    # It represent the uploaded file; counting lines
    count_line = 0
    while(True):
        line = fileitem.file.readline()
        if not line: break
        count_line = count_line + 1

Sometimes an error can interrupt the program while reading the content of the uploaded file (When a user clicks on Cancel Button or Back Button). FieldStorage class provides the done attribute to set to the value -1.

If we submit the form in the "old" format, the item will be instances of the class MiniFieldStorage. In this class, the list, file, and filename attributes are always None.

Generally, the form is submitted via POST and contains a query string with both FieldStorage and MiniStorage items.

Here, we are defining the FieldStorage attribute in the following table.

Attributes	Description
Name	It represents the field name.
Filename	It represents Client side filename.
File	It is a file(-like) object from which we can read data as bytes.
Value	It is a string type value. Use for file uploads, reads the file and returns bytes.
Type	It is used to display the content-type.
Header	It is a dictionary type object which contains all headers.

The FieldStorage instance uses the many built-in methods to manipulate the users' data. Below are a few FieldStorage's methods.

FieldStorage Methods:

Methods	Description
getfirst()	It returns the first value received.
getlist()	It returns the list of the received values.
getvalue()	It is a dictionary get() method.
keys()	It is dictionary keys() method
make_file()	It returns a readable and writable file.

Running First Python File as CGI

In this section, we will discuss how can run the CGI program over the server. Before doing this, we must ensure that our system has the following configuration-

Apache Server
Python

If you already have the XAMPP server in your system then you can skip this part.

Installing the XAMPP server

XAMPP stands for cross-platform, Apache, MySQL, PHP, and Perl, and it provides the localhost server to test or deploy websites. Generally, it gives the two essential components for its installation, first is - Apache that creates a local server and MySQL, which we can use a database.

Follow the below steps to install xampp.

Step - 1: Visit its official website (https://www.apachefriends.org/download.html) and download the latest version.

Step - 2: Once the download is complete, click on the run button.

Step - 3: Now, Click on the Next Button.

Step - 4: Next, it will display the xampp components. We can remove some of these but we will install all components of xampp, it won't affect our application.

Step - 6: Now, our set up is ready to install, start the installation by clicking the Next button.

It will take away on the webserver and start installing all packages and files.

Step - 7: Here, the download has completed. Select your desire language, and then, it will display the following dashboard of xampp.

Python Installation

To install the Python, visit out Python installation guide (how-to-install-python).

First CGI Program

We have created a new folder called example in xampp's htdocs folder. Then, we write a Python script, which includes the HTML tags. Let's see the following directory structure and the demo.py file.

demo.py

#C:\Users\DEVANSH SHARMA\AppData\Local\Programs\Python\Python37

print ("Content-Type : text/html\r\n\r\n")
# then come the rest hyper-text documents
print ("<html>")
print ("<head>")
print ("<title>My First CGI-Program </title>")
print ("<head>")
print ("<body>")
print ("
<h1>This is my CGI script </h1>
")
print ("</body>")
print ("</html>")

And, its directory structure as follow.

Type the localhost/example/demo.py into the web browser. It will display the following output.

Note - We need to start the Apache server then execute the CGI script. Our script demo.py will run on host 127.0.0.1 by default.

Let's understand another example of CGI script.

Example - 2:

#!/usr/bin/python3  
# Importing the 'cgi' module  
import cgi  
print("Content-type: text/htmlrnrn")  
print("<html><body>")  
print("  
   
<h1> Hello Program! </h1>  
   
")  
# Using the inbuilt methods  
form = cgi.FieldStorage()  
if form.getvalue("name")  
name = form.getvalue("name")  
print("  
   
<h1>Hello" +name+"! Thanks for using my script!</h1>  
   
")  
if form.getvalue("happy")  
print("  
 Yayy! I'm happy too!   
")  
if form.getvalue("sad")  
#!/usr/bin/python3  
# Importing the 'cgi' module  
import cgi  
print("Content-type: text/htmlrnrn")  
print("<html><body>")  
print("  
   
<h1> Hello Program! </h1>  
   
")  
# Using the inbuilt methods  
form = cgi.FieldStorage()  
if form.getvalue("name")  
name = form.getvalue("name")  
print("  
   
<h1>Hello" +name+"! Thanks for using my script!</h1>  
   
")  
if form.getvalue("happy")  
print("  
 Yayy! I'm happy too!   
")
if form.getvalue("sad")  
print("  
 Oh no! Why are you sad?   
")  
# Using HTML input and forms method  
print("  
   
<form method=post action=demo2.py>  
")  
print("  
Name: <input type=text name=name />  
")
print("<input type=checkbox name=happy /> Happy")  
print("<input type=checkbox name=sad /> Sad")  
print("<input type=submit value=Submit />")  
print("</form></body></html>")  

</body></html>

Structure of a Python CGI Program

Let's understand the following structure of the program.

The CGI script must contain two sections which separated by a blank line.
The header must be in the first section, and the second section will contain the kind of data that will be used during the execution of the script.

Legend of Syntaxes

When scripting a CGI program in Python, take note of the following commonly used syntaxes.

HTML Header

In the above program, the line Content-type:text/html\r\n\r\n is a portion of the HTTP, which we will use in our CGI programs.

HTTP Field Name: Field Content

For Example
Content-type: text/html\r\n\r\n

Sr.	Header	Description
1.	Content-type	It is a MIME string that is used to define the format of the file being returned.
2.	Expires: Date	It displays the valid date information.
3.	Location: URL	The URL that is returned by the server.
4.	Last-modified: Date	It displays the date of the last modification of the resource.
5.	Content-length: N	This information is used to report the estimated download time for a file.
6.	Set-Cookies: String	It is used to set the cookies by using string.

CGI Environment Variables

We should remember the following CGI environment variable along with the HTML syntax. Let's understand the commonly used CGI environment variables.

CONTENT_TYPE - It describes the data and type of content.
CONTENT_LENGHT - It defines the length of a query or information.
HTTP_COOKIE - It is used to return the cookie, which is set by the user in the current scenario.
HTTP_USER_AGENT - This variable is used to display the type of browser that the user is currently using.
REMOTE_HOST - It is used to describe the path of the CGI scripts.
PATH_INFO - This variable is used to define the path of the CGI script.
REMOTE_ADDR - We can define the IP address of the visitor by using it.
REQUEST_METHOD - It is used to make a request either via POST or GET.

Functions of Python CGI Programming

The CGI module provides the many functions to work with the cgi. We are defining a few important functions as follows.

parse(fp = None, environ = os.environ, keep_blanks_values = False, strict_parsing = False) - It is used to parse a query in the environment. We can also parse it using a file, the default for which is sys.stdin.
parse_qs(qs, keep_blank_values = False, strict _parsing = False) - While this is denigrated, Python uses it for urllib.parse.parse_qs() instead.
parse_qsl(qs, keep_blank_value = False, strict_parsing = False) - This is also denigrated, and maintains of for backward-compatibility.
parse_multipart(fb, pdict) - It is used to parse input of type multipart/form-data for file uploads. The first argument is the input file, and the second argument is a dictionary holding in the other parameters in the content-type header.
parse_header(string) - It is used to parse the header. It permits the MIME header into the main value and a dictionary of parameters.
test() - It is used to test a CGI script, and we can use it in our program. It will generally write minimal HTTP headers.
print_form(form) - It formats a form in HTML.
print_directory() - It formats the current directory in HTML.
escape(s, quote = False) - The escape() function is used to convert characters '<', '>', and '&' in the string's to HTML safe sequence.

Debugging CGI Scripts

First, we need to check the trivial installation error. Most of the time, the error occurs during the installation of the CGI script. In the beginning, we must follow the installation instructions and try installing a copy of this module file cgi.py as a CGI script.

Next, we can use test() function from the script. Type the following code with a single statement.

Advantages of CGI Programming

There are various advantages of using CGI programming. Below are some of its advantages.

They are language independent. We can use CGI programs with any programming languages.
The CGI programs can work on almost any web server and the portable.
They are portable.
The CGI programs can perform both simple and complex tasks; means they are fairly scalable.
The CGIs can increase the dynamic communication in the web applications.
The CGIs can also be profitable, if we use them in development; they reduce the development costs and maintenance costs.
The CGIs takes less time to process the requests.

Disadvantages of CGI

Consider the following disadvantages of CGI.

CGI programs are too complex and hard to debug.
When we initiate the program, the interpreter has to evaluate a CGI script in each initiation. The result is, creates a lot of traffic because are many requests from the side of the client-server.
CGI programs are quite vulnerable, as most of them are free and easily available without the server security.
CGI uses a lot of process time.
During the page load, the data doesn't store in the cache memory.
There are huge extensive codebases, most of it Perl.

Common Problems and Solutions

We can face problems during the implement the CGI script on the server. We have listed below the few common problems and their solutions.

First of all, check the installation instructions. Most of the problems occur during the installation of server. Follow the installation guide properly.
Check the HTTP server's log file. The tail -f logfile in a separated window may be valuable.
In the CGI, it is possible to display the progress report on the client's screen of running requests. Most HTTP servers save the output from the CGI script until the script is finished.
Before executing the file, check the syntax error in your script, following as python script.py.
If the script does not have any syntax error then import the library such as import cgitb; cgitb.enable() to the top of the script.
The absolute path must be included when importing the external program. The path is usually not set to a very useful value in a CGI script.
When reading or writing external files, make sure that they can be read or written by the user under which your CGI script will be running. This is authorized user id where the script file in which the web server is running or some specified userid for a web server.
It should be remembered that the CGI script must not set in set-uid It won't work on most systems, and also a security liability.