ping_test

DESCRIPTION

ping_test (pt.pl) is a cron driven user configurable perl script that pings an IP address or domain name at a user selected rate. An email is sent to one or more addresses when a user defineable number of packets in a row are dropped. Another email is sent when connectivity is restored. Ping times and dropped counts are stored in an RRDtool database. A webpage can be created at a user defineable rate showing all the statistics on an hourly, daily, weekly, monthly, and yearly rate. ping_test can automatically copy this webpage using scp to a remote webserver that is accessible using RSA public/private keys.

Since ping_test is written in perl it should be portable to other unix based systems. ping_test uses standard perl functionality. No additional perl modules are required. Use of RRDtool for generating graphics is optional. A text based webpage report is generated if RRDtool isn't available.

ping_test has several unique features. Its operation is controlled by a single "configuration file" for each IP or domain name being tested. Generated files are all uniquely named so a single copy of ping_test can be used to test multiple destinations. All files are stored in one directory making it easy to backup or move as needed.

The ping rate is user selected. The time of each ping is accurately calculated so there is very little jitter. For example, if the user configures the ping rate to be every 30 seconds, that will be the exact time.

The user configures the number of dropped packets required before an email is sent. The email contains a text summary of all the statistics.

ping_test does a traceroute which is included in the email. This is helpful in determining where the connectivity breakdown has occurred.

A record of when emails have been sent is included in the email. This shows the history of link stability.

ping_test maintains logfiles. These logfiles are automatically compressed and rotated weekly with the last four being retained. The output of each ping is stored in the logfile.

Using cron to start and monitor operation of ping_test means that once it is setup it rarely if ever needs to be touched again. It just keeps working and working. If the system it is running on gets rebooted, it simply starts back up.

At the time this file is being written the results of many ping tests are being posted to this site.

EXPLANATION OF RRDTOOL GRAPHICS

There are three RRD archives created to store data. Two types of data are stored: ping times and percentage of dropped packets. One archive stores individual data points for a week's worth of data. The second archive holds 4 week's worth of data points, but stores the maximum value of seven samples. The last archive stores 52 week's worth of data, but stores the maximum value of 52 samples. When RRD graphs are made the RRDtool automatically pulls data from the archive with the best fit. The RRDtool automatically does data point interpolation if the sample time isn't exactly the expected time for the archive. ping_test is designed to store data at exactly the expected RRD archive time. This means the values shown in the graphs are the exact values and not interpolations, which wouldn't give a true picture of what is going on. ping_test generates graphs for the last three hours, the last 24 hours, the last week, the last four weeks, and the last year. Click here to see an example of the graphics webpage created by ping_test. Clicking on the smaller graphics opens up a larger version with better detail.

Below is a "ping time" graph showing ping times for the last three hours. The y-axis is in milliseconds. The x-axis is time of day. The maximum ping time is about 820 milliseconds at 20:05. The average ping time hovers around 40 milliseconds.

Below are two graphs of the percentage of packets dropped, one for the last 24 hours and one for the last four weeks. ping_test resets the dropped count to zero at midnight for the daily graphs and at 00:00 hours on Sunday mornings for the weekly graphs. The yearly graphs actually never get reset. So, in the 24 hour graph below you see that the percentage loss goes to zero at 00:00 and stays at zero until about 07:30 when it jumps up to 0.400 percent. The percent value slowly glides down until about 19:40 when it jumps up again because another packet was dropped. In the 4 week graph you see the percent get reset to zero at 00:00 every Sunday morning. Week 32 had a higher percentage loss than the next weeks. The gliding down graphs make it very easy to see when packet losses occur.

INSTALLATION

Before ping_test will work there are a few configuration steps that must be done. Follow these steps on the machine where ping_test will run. ping_test has several different reporting methods: email, local webpage, and remote webpage. Each can be enabled separately.

1. Create a directory and copy pt.pl into this directory. A typical directory name is "ping_test". All generated files will be in this directory. Files are uniquely named so it is possible to have several instances of ping_test running from the same directory.

2. Edit the first line of pt.pl to point to where perl is installed on your system. The first time you run pt.pl from the command line it creates pt.conf, which must be edited to specify program locations for your system. A single instance of pt.conf is used for all ping_test running in the same directory. Error messages will be shown if programs can't be found. Edit pt.conf with the correct path information to the programs.

3. The example internet address "domain.com" is used for the rest of this setup help. Next invoke pt.pl with the internet address that will be tested.

         ./pt.pl domain.com

This creates the files shown below:

         domain.com -> pt.pl*
         domain.com.conf
         domain.com.logfile
         domain.com.pid
         domain.com.stats

Note that the symbolic link "domain.com" to "pt.pl" makes it possible to simply use "domain.com" as the shell command. This is a little easier than always having to type "pt.pl domain.com". The only place you must use "pt.pl domain.com" is in a crontab line.

4. Edit the "domain.com.conf" configuration file. This allows you to specify the email address where up/down notices are sent, the threshold number of dropped pings in a row before an email is sent, the ping rate, whether or not to generate local or remote webpages, whether or not to create graphics using the round robin database tool (see RRDtool at http://oss.oetiker.ch/rrdtool) and options used by scp and ssh to transfer files to a remote webserver if desired. Setting up for remote webserver updates is described at the end of this setup help.

5. The next step is to manually start up ping_test while tailing the logfile to verify proper operation. The logfile is the default location for all program output. In one shell use this command to tail the logfile:

       tail -f /path/to/ping_test/domain.com.logfile

In another shell use the following command to start up ping_test:

       /path/to/ping_test/domain.com

If all is working well you will see lots of information. Look for any error messages and fix the problems as needed. You can set a debug flag that will force ping_test to simulate dropped packets. See the debug help page for information (pt.pl -hd).

6. Once manual operation is confirmed it is time to create a crontab entry so the ping_test will be started automatically. A typical crontab entry looks like this:

     0,15,30,45 * * * * /path/to/ping_test/pt.pl domain.com

Note that "domain.com" is used as an argument to "pt.pl". This is the correct syntax for the crontab entry. The above crontab line checks every 15 minutes to see that ping_test is running and restarts it if the ping_test pid isn't active. When everything is working correctly the cron task simply exits.

7. You can kill all processes associated with a ping_test with either of these commands:

        /path/to/ping_test/pt.pl domain.com -k
        /path/to/ping_test/domain.com -k

8. You can restart a ping_test with either of these commands:

        /path/to/ping_test/pt.pl domain.com -restart
        /path/to/ping_test/domain.com -restart

This is useful when you make changes to the domain.com.conf file and need to restart ping_test right away. This stops a current ping_test and restarts it thereby reading the domain.com.conf file.

SETTING UP AND TESTING REMOTE WEBSERVER UPDATES

The last section in domain.com.conf sets up options needed by SSH and SCP to securely copy files to a remote webserver. You must have an account on the webserver machine which is accessible using SSH with RSA public/private keys.

1. Use the "ssh-keygen" utility to generate an RSA public/private key pair. Name the key something like "id_rsa.machine_name" so it is easily identifiable. Do not give the key a pass phrase.

2. Copy public RSA key to your ~/.ssh/authorized_keys file on the remote webserver machine.

3. Put the private RSA key in the /path/to/ping_test directory. Change to the ping_test directory and set permissions to owner access only. The RSA private key file must only be readable by the owner of the file, ie. "chmod 400 filename". The directory must only be writeable by the owner, ie. 700. Use chmod to set permissions accordingly.

4. Modify the domain.com.conf file with the correct values for the following:

         WEBPAGE_REMOTE_DIR   /path/on/webserver/to/files
         HostName             webserver_WAN_name
         Port                 22
         UserKnownHostsFile   /path/to/ping_test/domain.com.known_hosts
         User                 user_login_name_on_webserver
         IdentityFile         /path/to/ping_test/id_rsa.private_key_name

After modifying the domain.com.conf file you must restart the ping_test so that the domain.com.ssh file is recreated with the new information.

5. Verify the ssh connection can be made. If everything is setup correctly you should be able to start a shell on the webserver without entering any password. In a shell use this command to connect to the webserver machine.

        ssh -F domain.com.ssh webpage

The above command verifies the settings in the domain.com.conf file. You will be warned that the ssh server's key doesn't exist and you will be asked if you want to add it. Answer yes. Once this key is in the known_hosts file you won't be asked for it anymore. Type exit to return to the ssh client shell.

If errors occur, use the following command to make a verbose connection to the webserver machine. Carefully look through the debug output to see what the problem is:

        ssh -v -F domain.com.ssh webpage

6. Webpages and graphics will now automatically be copied over at the rate specified by the WEBPAGE_UPDATE_RATE in domain.com.conf. Graphics are stored in WEBPAGE_REMOTE_DIR/png on the remote webserver and in WEBPAGE_LOCAL_DIR/png locally. These directories are made automatically if they don't exist.

WEBPAGE GRAPHICS SUMMARY PAGES

If many ping_tests are copying their webpages to the same webserver directory it may be desireable to see all the graphics at the same time. There is an option "-ms n" to pt.pl that creates summary webpages. The way this works is to copy pt.pl into the WEBPAGE_REMOTE_DIR/png or the WEBPAGE_LOCAL_DIR/png directory. In that directory invoke "pt.pl -ms n" where n is the number of seconds between auto webpage refreshes. Enter 0 for n if auto refreshes are not desired. Invoking pt.pl in this way creates a set of summary webpages in the WEBPAGE_REMOTE_DIR and WEPAGE_LOCAL_DIR directories. The summary pages have names like "summary_3hrs.html". This webpage displays all the domain.com.3hrs.png graphs on one page. This makes it easy to compare results and is especially useful if domain.com is being tested from several different locations. "pt.pl -ms n" uses whatever png files are in the png directory at the time to create the summary pages. If tests are added or deleted then the summary webpage files will need to be remade.

You can see an example of a 3hrs summary page by clicking here.

HISTORY

ping_test started its existence in 2003. It was first written as a hideous tcsh script. The tcsh version was 1.x. In March 2004 it was ported to bash and mostly rewritten. This was a huge improvement in reliability and maintainability. The bash version was 2.x. When support for cygwin was added in April 2005 the version number was changed to 3.x because it was such an involved task. In Feb 2006 ping_test was ported to perl and became version 4.x. Graph generation using RRDtool was added in June 2006. Lots of debugging and tweaking has taken place to get ping_test to its current form. It has been extensively tested on Linux, Solaris, and Cygwin. It is very stable on Linux and Solaris. Cygwin works okay, but it is less stable.

SUPPORTED PLATFORMS:

This script has been used successfully on the following operating systems:

    Solaris 7, 8, 9
    Linux 2.x on Mandrake 9.1, Yellow Dog 4.1, Gentoo
    Cygwin 1.5.x on Windows 2000 and Windows XP

Copyright: 2003-2006 by John R Larsen - john@larsen-family.us
http://pingtest.sourceforge.net
Released under the same Artistic license as perl