Chapter 19 Using Web

1. Lynx Program

Like the Internet, the Lynx program is a very old text-based web program. Lynx program allows users to use Wbe on the terminal. Through the Lynx program in Shell Script, web page data can be received in text-based format and can be processed and used in Script.

​

Installing Lynx

1) Download the latest package to install from the site below.

​

Site: LYNX – The Text Web-Browser (invisible-island.net)

​

2) Unzip the downloaded file in the working directory as shown below.

tar -zxvf lynx2.8.6.tar.gz

​

3) Change the working directory to the unzipped directory.​

cd lynx2.8.6

​

4) Execute the setting script.

./configure

​

5) Execute the make command to compile the source code and create an executable file.

​

6) Copy the executable lynx file as the root user to the common directory in the user's PATH. Usually, the common directory on the Relux system is /usr/local/bin .

​

lynx command line.

The lynx command has a wide variety of information that can be retrieved from a remote web. There are three types of information transmitted to the user's browser in the web.

​

■ HTTP headers: Includes basic information such as the type of data sent from the web server

■ Cookies: Store visit history

■ HTML content: The part where the actual data of HTML is saved. ​

​

The lynx program shows the actual HTML content in 3 modes.

■ Text graphics display mode for terminal sessions using the curses graphics library

■ Dump raw data from a web page to a text file.

■ Dumping raw HTML source code from web pages to text files

​

Basic command format in lynx​

■lynx options URL

​

​

In url, you can enter the address of the web site you want to access, and in option you can define options that define how to work with the web server. For example, if you want to post data on the web, you can use the -post -data option.

​

Using the lync configuration file

lynx has a number of frequently used options, which can be cumbersome to type each time on the command line. For this, you can use the frequently used options by pre-defining them in the configuration file.

​

The location of the file is as follows.

/usr/local/lib/lynx.cfg

​

How to use the configuration file

PARAMETER:value

​

​

There are many command parameters in lynx, please refer to the table below.

​

Lync Command Parameter

​

​

Below are the parameters that define the proxy server.

http proxy:http://some.server.dom:port/ https proxy:http://some.server.dom:port/ ftp proxy:http://some.server.dom:port/ gopher proxy:http://some.server.dom:port/ news proxy:http://some.server.dom:port/ newspost proxy:http://some.server.dom:port/ newsreply proxy:http://some.server.dom:port/ snews proxy:http://some.server.dom:port/ snewspost proxy:http://some.server.dom:port/ snewsreply proxy:http://some.server.dom:port/ nntp proxy:http://some.server.dom:port/ wais proxy:http://some.server.dom:port/ finger proxy:http://some.server.dom:port/ cso proxy:http://some.server.dom:port/ no proxy:host.domain.dom

​

​

Lynx environment variables

You can customize the Lynx program to your liking through command-line options and configuration files. If you do not have access to the configuration file, you can customize it by setting environment variables.

​

​

​

The table below shows the environment variables of lynx.

​

​

Capturing data with Lynx

When using Lynx in scripts, the most common purpose of using it is to get a specific piece of information from a web page. The way to make this possible is called screen scraping, and the way to use it is to use the -dump option, and it outputs the data displayed on the web page to standard STDOUT.

​

​The usage and output results are as follows.

$ lynx -dump http://localhost/RecipeCenter/ ​ The Recipe Center "Just like mom use to make" ​ Welcome [1]Home [2]Login to post [3]Register for free login [4]Post a new recipe ​ ​

Let's look at a more detailed usage example. Let's see how to extract weather information for a specific area from Yahoo. First, each region uses a specific url to separate data and display it. If you look at how to find weather information for Illinois in Chicago, redirect the lynx command result to a file, This is how to extract the desired data from the file using sed or gwak editor.

lynx -dump http://weather.yahoo.com/forecast/USIL0225.html ​ Current conditions as of 1:54 pm EDT Light Drizzle Feels Like: 50◦ Barometer: 29.34 in and falling Humidity: 93% Visibility: 4 mi Dewpoint: 48◦ Wind: S 16 mph

In the example above, you can see that the value of each weather item is displayed below that item. Let's process this and make a script that displays only the desired data.

$ cat sedcond /Current conditions/{ n n p } $ ​ $ cat sedtemp /Feels Like:/{ n p } $ ​ ​ ​ $ cat weather #!/bin/bash # extract the current weather for Chicago, IL URL="http://weather.yahoo.com/forecast/USIL0225.html" LYNX=`which lynx` TMPFILE=`mktemp tmpXXXXXX` ​ $LYNX -dump $URL > $TMPFILE ​ conditions=`cat $TMPFILE | sed -n -f sedcond` temp=`cat $TMPFILE | sed -n -f sedtemp` rm -f $TMPFILE echo "Current conditions: $conditions" echo The current temp outside is: $temp ​ ​ $ ./weather Current conditions: Light Rain The current temp outside is: 49◦ $

​2. cURL program

The popularity of the LYNX program led to the birth of a program called cURL. This program allows you to transfer files using a specific URL from the command line. ​ Currently supported protocols include FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, Telnet, DICT, LDAP, LDAPS, etc. ​ Now, let's see how to install and use cURL.

​

Installing cURL

In most Linux systems, it is installed by default, but if not, you can install cURL according to the procedure below.

​

1) Download from the address below.

Site: curl.haxx.se

​

2) Unzip it.

tar -zxvf Downloaded file name .tar.gz

​

3) Move to the appropriate directory. ​

cd curl-7.18.0

​

4) Execute the configuration file.

./configure

​

​

5) Execute make to compile the source.

make

​

6) It copies executable files as root user to the common working path. ​

/usr/local/bin

​

​

Using cURL

cURL basically returns the entire HTML code to STDOUT. Let's look at an example below.

$ curl http://localhost/RecipeCenter/ <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <link rel="stylesheet" type="text/css" href="mystyle.css" /> <link rel="stylesheet" media="print" type="text/css" href="print.css" /> <title>The Recipe Center</title> </head> ​ ​ <body> <table width="100%" border="0"> ​ <tr> <td id="header" height="90" colspan="3"> <h1><br> The Recipe Center</h1> <h4> <em> "Just like mom use to make"</em></h4></td> </tr> <tr> <td id="nav" width="15%" valign="top"> <table width="100%" cellpadding="3"> <tr> <td><h3>Welcome</h3></td> </tr> <tr> [ listing truncated ]

​

cURL has as many options as Lynx does. Please refer to the table below for available parameters.

​

​

​

Let's look at an example of downloading the latest IOS from WEB as a simple usage example of cURL. Please refer to the script below.

$ cat downld #!/bin/bash # download latest cURL file automatically ​ curl -s -o /home/rich/curl-7.18.0.tar.gz http://curl.haxx.se/downloads/curl-7.18.0.tar.gz $

​

Networking with zsh

zsh is a shell that provides many functions in the form of modules. Among them, let's see how to do networking using TCP modul.

​

​

TCP Module

Installation is simple. Just execute the command below in the shell.

% zmodload zsh/net/tcp

%

​

​

After loading the TCP module, it is used using the command, and how to use the command is:

​

ztcp [-acflLtv] [ -d fd] [args]

​

The details of the options are as follows.

■ -a: Allow new connections

■ -c: Terminate an existing connection

■ -d: to use a specific file descriptor for the connection

■ -f: force connection to close the connection

■ -l: Open a new socket for listening

■ -L: List currently connected sockets

■ -t: Terminate if there are no pending connections

■ -v:Display detailed information about the connection

​

The ztcp program uses FD to interact with open TCP connections. By default, zsh uses the environment variable $RESULT to refer to FD. All it needs is to send the data to the file descriptor specified in the $RESULT variable, and the TCP module forwards it to the remote host. Similarly, when the remote host sends data, all that is needed is to read the data from the FD specified in the $RESULT variable.

​

The following is the procedure for communicating between the server and the client.

Now, let's take a look at the sample scripts of server and client.

​ Server side script % cat server #!/bin/zsh # zsh TCP server script zmodload zsh/net/tcp ​ ztcp -l 5150 fd=$REPLY echo "Waiting for a client..." ztcp -a $fd clientfd=$REPLY echo "client connected" echo "Welcome to my server" >& $clientfd while [ 1 ] do read line <& $clientfd if [[ $line = "exit" ]] then break else echo Received: $line echo $line >& $clientfd fi done echo "Client disconnected session" ztcp -c $fd ztcp -c $clientfd % ​ ​ Clien side Script % cat client #!/bin/zsh # zsh TCP client program zmodload zsh/net/tcp ztcp localhost 5150 hostfd=$REPLY read line <& $hostfd echo $line while [ 1 ] do echo -n "Enter text: " read phrase echo Sending $phrase to remote host... echo $phrase >& $hostfd if [[ $phrase = "exit" ]] then break fi read line <& $hostfd echo " Received: $line" done ztcp -c $hostfd % ​ ​

Create Server and Client scripts using TCP module and learn various options.