Posts Tagged PERL

Error_reporting in PHP

I have rich experience in programming PHP and PERL. In Perl, I always add the following two lines to my scripts:

use strict;  user warning;

However, I could not find similar things in PHP. When I am working with a complicated PHP web application project, I encountered a lot of times that wrote a variable name in one way and use call it in a wrong spelling. That makes me really frustrating. Sometimes, took me a long time to figure out what was wrong. Then I started to search the internet and tried to find a solution to reduce the frustrating moments.  End up I found post with the following question:

I’m getting the following error PHP Notice: Undefined index: HTTP_REFERER in C:\Inetpub\cgi-bin\search5.php on line 168 on first call..
i’m just trying to get the value of this $_SERVER[‘HTTP_REFERER’]

Aany solution for this?

I got two useful information from the follow-up posts

1) Check the variable is set or not: isset() function. For example, isset($arr[‘a’]);

2) Switch on the PHP error reporting mechanism. There are two ways to do this. In php.ini file,  uncomment the following line:

error_reporting E_ALL

Or on the top of each PHP file, include the following code:


There is also a display_errors directive which allows you to display errors, warnings and notices in your browser or have them written to the server’s error log only.

By using the above method, I can get rid of all unnecessary warnings and errors I made in the PHP applications.


Tags: , , ,

PERL processes files and related functions

Command line parameters

If you want to know how many command line arguments the program gets, you can use $#ARGV because ARGV is an predefined array in Perl. Here is code snippet to use this array.

if ($#ARGV != 1)
print "Usage: you have provide two inputs";
my $mypath = $ARGV[0];

Change folder

It is as simple as simply use cd command in OS.

chdir($mypath) || die "$!";

When you use folder path in the PERL variable, please remember use \\ in DOS or Windows system to represent path, for example “.\\temp\\” represent the subfolder “temp” in the current folder. Otherwise you will get very strange errors when you run the PERL script.

Read the whole content in a file

The key command is “local $/=undef;”. See the code snippet to learn the detail.

open(DATA, "< $filename")
|| die "Failed to open the file $filename\n$!\n";

local $/ = undef;
my $content = <DATA>;


Process text content

After you read the whole content in a file to a variable, you have to process it. If you have to process it line by line, you can use split function to break them into an array, like the following.

my @lines = split(/\n/,$content);

If you want to get rid of the first line, you can use the following command to achieve it.


If you want to break the content in one line by space, you can use the following command.

my @temps = split(/\s+/,$temp);

The key is the regular expression \s+, which stands for one or more space.


Tags: , , , ,

Convert ip address from dotted quad notation to integer

Normally we see the IP address of our computers is a dotted quad, such as Frequently, IP address is stored in database as integer to save space and provide high performance. How to convert the IP address from a dotted quad to an integer? Let us understand the algorithm of the conversion. Once we understand the underlying theory, it will be much easier to implement the conversion algorithm in any language. All information are from the internet. I do not claim anything is developed by myself.

The algorithm

To convert dotted quad (A.B.C.D) to a number:
+ 256 * C
+ 256 * 256 * B
+ 256 * 256 * 256 * A

To convert number (X) back to dotted quad:
temp = X / 256
D = 256 * (temp – INT(temp))
temp = (INT (temp)) / 256
C = 256 * (temp – INT(temp))
temp = (INT (temp)) / 256
B = 256 * (temp – INT(temp))
A = INT (temp)

You may need FLOOR() instead of INT() in some language. There is a chance we encounter the endian-ness error. In that case swap A<->D and B<->C in the calculations.


Here’s one way, using Regexp::Common, although it’s unlikely to be the most efficient:
use Regexp::Common;
$ip =~ s(
join ‘.’, map { hex } $2, $3, $4, $5

You could also do it as follows, although it’s still probably not as fast as the pack/unpack method:
join ‘.’, map { hex } $ip =~ /../g;

What would be the most efficient way to convert an IP address from hexadecimal to dotted quad notation?
E.g. from 7F000001 to

I came up with:

$ip = ‘7F000001’;

use Socket;
inet_ntoa( pack( “N”, hex( $ip ) ) )

Q&A > data formatting > Converting an IP from hexadecimal to dotted-quad contributed by merlyn

join ‘.’, unpack “C*”, pack “H*”, $ip;


ip2long and long2ip in PHP can be used to convert ip address.

* int_oct($ip)
* Convert INTeger rep of IP to octal (dotted quad)
function int_oct($ip) {
/* Set variable to float */
settype($ip, float);

/* FIX for silly PHP integer syndrome */
$fix = 0;
if($ip > 2147483647) $fix = 16777216;

if(is_numeric($ip)) {
$ip / 16777216,
(($ip % 16777216) + $fix) / 65536,
(($ip % 65536) + $fix / 256) / 256,
($ip % 256) + $fix / 256 / 256
else {

Here are some tricks to convert from a “dotted” IP address to a LONG int, and backwards. This is very useful because accessing an IP addy in a database table is very much faster if it’s stored as a BIGINT rather than in characters.


IP as BIGINT read from db back to dotted form:

Keep in mind, PHP integer operators are INTEGER — not long. Also, since there is no integer divide in PHP, we save a couple of S-L-O-W floor (
)’s by doing bitshifts. We must use floor(/) for $ipArr[0] because though $ipVal is stored as a long value, $ipVal >> 24 will operate on a truncated, integer value of $ipVal! $ipVint is, however, a nice integer, so
we can enjoy the bitshifts.
floor( $ipVal / 0x1000000) );
$ipVint = $ipVal-($ipArr[0]*0x1000000); // for clarity
$ipArr[1] = ($ipVint & 0xFF0000) >> 16;
$ipArr[2] = ($ipVint & 0xFF00 ) >> 8;
$ipArr[3] = $ipVint & 0xFF;
$ipDotted = implode(‘.’, $ipArr);

Here are some tricks to convert from a “dotted” IP address to a LONG int, and backwards. This is very useful because accessing an IP addy in a database table is very much faster if it’s stored as a BIGINT rather than in characters.


This can be written in a bit more efficient way:
$ipArr = explode(‘.’,$_SERVER[‘REMOTE_ADDR’]);
$ip = $ipArr[0]<<24
+ $ipArr[1]<<16
+ $ipArr[2] <<8 + $ipArr[3] ; ?>

shift is more cheaper.


A note about converting IP addresses for storage in database. For MySQL, this is unnecessary as it has built in support via the INET functions. Also, there is no need to use BIGINT. UNSIGNED INT is, at 4 bytes, the perfect size for holding an IP (column must be defined as UNSIGNED). This can basically halve the storage size, as BIGINT is an 8 byte data type.

INET_ATON() converts a dotted IP string to INT:

INET_NTOA() converts an INT to dotted IP string:
returns ‘’


Tags: , , , , ,

A PERL script to get Twitter user information

If you are a Twitter user and do business on Twitter, you probably think about how I can target to a group of Twitter users, such as living in certain region or having similar interests. Can we screen Twitter users and target to a given population of Twitter users? Answer is absolutely positive: yes, we can. One simple solution is to write a PERL script to download Twitter user information and store them into database first. Then we can conduct different sort of queries to achieve our goal. The following PERL script can download Twitter user information, extract user information and then store in MySQL database. Here is the source code.

require 5.6.0;
use strict;
use warnings;
use DBI;
use Data::Dumper;
use XML::Simple;

my $dbh;
my $sql;
my $sth;
my $dbname = "twitter_world";
my $tablename = "users";
my $xmlname = "tid.xml";


my $i = 100;
while (1==1)



sub process {

	my $tid = $_[0];
	system("curl$tid -o tid.xml");

	# create an XML object
	my $xml = new XML::Simple;

	# read the RSS feed file
	my $data = $xml->XMLin($xmlname);

	#print Dumper(\$data); 
	if ($data->{error}) { return; }
	my $location        = checkVar($data->{location});
	my $time_zone       = checkVar($data->{time_zone});
	if (($location eq "") && ($time_zone eq "")) { return; }
	my $id              = checkVar($data->{id});
	my $name            = checkVar($data->{name});
	my $screen_name     = checkVar($data->{screen_name});
	my $description     = checkVar($data->{description});
	my $url             = checkVar($data->{url});
	my $followers_count = checkVar($data->{followers_count});
	#my $friends_count   = checkVar($data->{friends_count});
	#my $created_at      = checkVar($data->{created_at});
	#my $utc_offset      = checkVar($data->{utc_offset});
	my $sql = "insert into users (id, name, screen_name, location, description, url, time_zone) ".
	          "values ($id, $dbh->quote($name), $dbh->quote($screen_name), $dbh->quote($location), $dbh->quote($description), $dbh->quote($url), $dbh->quote($time_zone));";
	#print "$sql\n";

sub checkVar
	my $var = $_[0];
	if (ref($var) eq 'HASH')
		$var = "";
	return $var;

# execute a SQL statement
sub executesql {
	my $sth = $dbh->prepare($_[0]);
	my $nrec = $sth->execute();
	return $nrec;

# connect to database and generate a handle
sub getConnected {
# return the database handle object to the caller

	# Set the parameter values for the connection
	my $host="host address";
	my $connectionInfo = "DBI:mysql:$dbname;$host";
	my $databaseUser = "mysql database username";
	my $databasePw = "mysql database password";

	# Connect to the database
	# Note this connection can be used to 
	# execute more than one statement
	# on any number of tables in the database
	$dbh = DBI->connect($connectionInfo, $databaseUser, 
	    $databasePw) || die "Connect failed: $DBI::errstr\n";	    

# disconnect the database handle
sub disConnected {


Tags: , , , ,

SQLite database engine and sample code in C#

The definition in SQLite official site is the following: “SQLite is a software library that implements a self-contained, serverless, zero-configuration, transactional SQL database engine. SQLite is the most widely deployed SQL database engine in the world.” So, how can we use SQLite engine to develop standalone desktop database application in Windows environments. At SQLite official site, you can download all C source of the SQLite library. Nowadays people develop .NET applications by using C#, VB.NET, C++ etc. I usually use C# to develop Windows applications. How can we utilize SQLite library in .NET environments? In another words, does a SQLite .net interface exist? Yes, it does. System.Data.SQLite, an open source ADO.NET provider for the SQLite database engine, provides all the functionality you need to operate SQLite databases. What you need to do is to download latest version of System.Data.SQLite at Run the setup file and install SQLite library and the .NET interface library into your PC. Then you can use this library to develop standalone high standard database application. To operate the database you can use commands of SQL language. Pretty a amazing. I wrote a PERL script running in Linux system to operate MySQL databases on MySQL server. Later on, I ported this PERL script to C#. I did not need to change any SQL statements used in the PERL script. Some of them are really complicated.

In order to give your some guideline to start using System.Data.SQLite library, I am going to share a simplified framework, a C# class, with you. You can modify it to use it in your project freely.

 * Author: Zhanshan Dong
 * Created: 4/3/2009
 * Modified: 9/18/2009

using System;
using System.Windows.Forms;
using System.Data.SQLite;
using System.IO;
using System.Collections.Generic;

namespace TwitterGIFT
	/// Description of Class1.
	public class GiftSQLite
		private SQLiteConnection Conn;
		private SQLiteCommand Cmd;
		private string dbFilename = "";
		public GiftSQLite(string userName)
			username = userName;
			dbFilename = userName + ".db";
			string FullPath = Application.StartupPath + "\\" + dbFilename;
			if (Application.StartupPath.Substring(Application.StartupPath.Length - 1,1) == "\\")
			    FullPath = Application.StartupPath + dbFilename;
			bool hasDBFile = File.Exists(FullPath);
			Cmd = CreateCMD();
			if (! hasDBFile ) 

		private void ConnectDB()
			Conn = new SQLiteConnection();
			Conn.ConnectionString = "Data Source=" + username + ".db;New=True;Compress=True;Synchronous=Off";
		private SQLiteCommand CreateCMD()
			SQLiteCommand cmd = new SQLiteCommand();
			return Conn.CreateCommand();

		private void disposeCMD(SQLiteCommand cmd)
		private void ExecuteNonQuerySQL(string SQL) {
			Cmd.CommandText = SQL;

		private SQLiteDataReader ExecuteQuerySQL(string SQL, SQLiteCommand cmd) {
			cmd.CommandText = SQL;
			return cmd.ExecuteReader();
		private void CreateTables()
			// create a table
			ExecuteNonQuerySQL("CREATE TABLE tablename (field1 integer primary key, ...)");

		private int getRowCount(string tablename)
			SQLiteCommand cmd = CreateCMD();
			SQLiteDataReader dbReader = ExecuteQuerySQL("SELECT count(id_"+ tablename + ") FROM "+ tablename,cmd); 
			int rowCount = 0;
			if (dbReader.HasRows) { dbReader.Read(); rowCount = dbReader.GetInt32(0); }
			return rowCount;

		private void processAllRows()
			SQLiteCommand cmd = CreateCMD();
			SQLiteDataReader dbReader = ExecuteQuerySQL("SELECT * FROM tablename",cmd);
			if (dbReader.HasRows) 
				while (dbReader.Read())
					int id = dbReader.GetInt32(0);

Besides the functionality in your application to do database operation, you do need SQLite database client tools to browse and manage your database. There are plenty of tools available. I tested a very simple one from RazorSQL provides an array of tools for editing, browsing, querying and managing SQLite database. TwitterGIFT is a Windows application we developed that utilized the SQLite engine to handle database.


Tags: , , , , ,