Ivan Villareal » Perl script to manage virtual hosts – Ivan Villareal

Perl script to manage virtual hosts

Ivan Villareal — Wed, 08 Jun 2011 19:54:57 +0000

I did this script a while back, and I’ve been using it on gentoo, ubuntu and centos distros with a few changes, this can be extended to add new functionality, and it may contain bugs, but so far it has worked for me pretty well, I can add/remove virtual hosts very quickly.

To use this script you only need the packages File::Path and Getopt::Long; both of them are already on newer distros, just put the script somewhere I have it in /usr/local/bin and call it with sudo.

ivan@orion:/var/www/vhosts$ sudo vhost-updater.pl --add --domain test.orion

This is the output:

Creating docroot dir: /var/www/vhosts/test.orion/public_html 
Creating log dir: /var/www/vhosts/test.orion/logs 
Site File: /etc/apache2/sites-available/test.orion 
Creating Vhost... 
Adding host test.orion 
Run '/etc/init.d/apache2 reload' to activate new configuration!
Restarting apache 
 * Restarting web server apache2
 ... waiting    ...done.

Once it has finished, point your browser to http://test.orion, and you’re done.

Enjoy!

#!/usr/bin/perl -w
 
#############################################################
# The purpose of this script is to add/remove				# 
# virtual hosts easily, this script runs 					#
# on Ubuntu/Debian withouth modifications. 					#
#         													#	
#         													#	
# Author Ivan Villareal ivaano@gmail.com					#
#         													#	
#############################################################
use strict;
use File::Path qw(mkpath rmtree);
use Getopt::Long;
 
 
our $ipAddress        = '127.0.0.1';
our $apacheConfigDir  = '/etc/apache2';
our $sitesAvailable   = 'sites-available';
 
our $docRootPrefix    = '/var/www/vhosts';
our $docRoot          = 'public_html';
our $logsDir          = 'logs';
 
 
 
my $del = '';
my $add = '';
my $domain = '';
 
if (getpwuid( $< ) ne 'root') {
	print "Script needs root privileges \n";
	exit();
}
 
unless (GetOptions (
		'del' => \$del, 
		'add' => \$add, 
		'domain=s' => \$domain) or usage()) {
    usage();
}
 
#print $paramResults;
if ($add || $del) {
    if ($domain) {
        if ($add) {
            createVhost($domain);
        } elsif ($del) {
            deleteVhost($domain);
        }
    } else {
        usage();
    }
} else {
    usage();
}
 
 
sub usage {
    print < <USAGE
This program will add or remove apache virtual hosts.
 
usage: vhost-updater.pl [--add | --del] --domain newhost.tld 
USAGE
}
 
sub returnVhostPaths
{
    my $vhost = shift;
    my @dir = split(/\//, $docRootPrefix);
    my %res;
 
    push(@dir, $vhost);
 
    my $hostDir = join('/', @dir);
    $res{'docRoot'} = $hostDir . '/' . $docRoot;
    $res{'logsDir'} = $hostDir . '/' . $logsDir;
	$res{'hostDir'} = $hostDir;
    #todo dir validation
    @dir = split(/\//, $apacheConfigDir);
    push(@dir, $sitesAvailable);
    push(@dir, $vhost);
    $res{'apacheConfig'} = join('/', @dir);
 
    return %res;   
}
 
sub createVhost {
    my $vhost = shift;
    #first create the docRoot
    my %vhostInfo = returnVhostPaths($vhost);
 
    informOut("Creating docroot dir: $vhostInfo{'docRoot'}");
    mkpath($vhostInfo{'docRoot'});
	my $user = getlogin();
	my $uid  = getpwnam($user);
	my $gid  = getgrnam($user);
	chown $uid, $gid, $vhostInfo{'hostDir'};
	chown $uid, $gid, $vhostInfo{'docRoot'};
    informOut("Creating log dir: $vhostInfo{'logsDir'}");
    mkpath($vhostInfo{'logsDir'});
 
 
    informOut("Site File: $vhostInfo{'apacheConfig'}");
 
    my $vhostContent = << "EOF";

    ServerName $vhost
    DocumentRoot $vhostInfo{'docRoot'}
    
        Options Indexes FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
    
        ErrorLog $vhostInfo{'logsDir'}/error_log
        CustomLog $vhostInfo{'logsDir'}/access_log "%h %l %u %t \\"%r\\" %>s %b \\"%{Referer}i\\" \\"%{User-agent}i\\""
        LogLevel debug
 
 
EOF
    informOut("Creating Vhost...");
    open FILE, ">", $vhostInfo{'apacheConfig'} or die $!;
    print FILE $vhostContent;
    close FILE;
 
    informOut("Adding host $vhost");
    open FILE, ">>", '/etc/hosts' or die $!;
    print FILE $ipAddress ."\t". $vhost ."\n";
    close FILE;
 
    my $output = `/usr/sbin/a2ensite $vhost`;
    print $output;
 
    restartApache();
    #print $vhostConten t;
}
 
sub restartApache
{
    informOut("Restarting apache");
    my $output = `/etc/init.d/apache2 restart`;
    print $output; 
}
 
sub deleteVhost
{
    my $vhost = shift;
     my %vhostInfo = returnVhostPaths($vhost);
 
    informOut("Removing $vhost from hosts file");
    open IN, '< ', '/etc/hosts' or die $!;
    my @hostsFile = ;
    close IN;
 
    my @contents = grep(!/^127.0.0.1\t$vhost/, @hostsFile);
 
    open FILE, ">", '/etc/hosts' or die $!;
    print FILE @contents;
    close FILE;
 
    my $output = `/usr/sbin/a2dissite $vhost`;
    print $output;
 
    informOut("Removing  $vhostInfo{'apacheConfig'} file");
    unlink($vhostInfo{'apacheConfig'});
 
    restartApache();
 
    print " manually remove $vhostInfo{'docRoot'}... \n";
 
}
 
sub informOut {
    my $message = shift;
    print "$message \n";
}

Script to copy data from MySQL to postgreSQL

Ivan Villareal — Sat, 02 Apr 2011 12:28:32 +0000

This is a perl script I’ve used to move data from mysql 5.1 to postgres 9, this script doesn’t create a schema, it only grabs the data from mysql and try to put it on the same table in postgres.

The only requirement is DBI for mysql and postgres.

 
#!/usr/bin/perl
#
# this script moves all the data from mysql to postgres
# modules required are:
#
# http://search.cpan.org/~timb/DBI-1.616/DBI.pm
#
# please change the dsn, srcUser/srcPass and dstUser/Pass variables.
 
use DBI;
use strict;
 
my $srcUser   = 'ivan';
my $srcPass   = '123';
my $srcDbName = 'datatocopy';
my $srcDbHost = 'mini';
 
my $dstUser  = 'ivan';
my $dstPass  = '456';
my $dstDbName = 'migrateddata';
my $dstDbHost = '192.168.1.145';
 
### do not modify ##
 
my $srcDb  = DBI->connect("dbi:mysql:database=$srcDbName;host=$srcDbHost", $srcUser, $srcPass) or die;
my $dstDb  = DBI->connect("dbi:Pg:database=$dstDbName;host=$dstDbHost", $dstUser, $dstPass) or die;
 
####main########
print "Starting to move data from $srcDbHost to $dstDbHost\n";
$srcDb->do("SET NAMES \'UTF8\'"); 
$dstDb->do("SET CLIENT_ENCODING=\'UTF8\'") or die;
 
 
my $srcTables = $srcDb->selectall_arrayref("SHOW TABLES");
my $dstTables = $dstDb->selectall_hashref("SELECT * FROM pg_tables WHERE NOT tablename ~\'^(pg_|sql_)\'","tablename");
 
my $seqlist = $dstDb->selectall_arrayref("SELECT RELNAME FROM pg_class WHERE relkind=\'S\'");
my $seqmap;
foreach my $s(@$seqlist)  {
    my $seqname = $s->[0];
    if ($seqname=~/^(.*)_([a-z]+)_([a-z]+$)/) {   
        my $item;
        $item->{seq}=$seqname;
        $item->{field}=$2;
        my $table = $1; 
        if (exists $dstTables->{$table}) {   
            push @{$seqmap->{$table}},$item;
        } else {   
            print STDERR "WARN: cannot find table for sequence $seqname\n";
        }   
    }   
}
$dstDb->{AutoCommit}=0;
my $totalRowCount = 0;
foreach my $t(@$srcTables) {  
    my $table = $t->[0];
    unless (exists $dstTables->{$table}) {
        print STDERR "WARN: table $table does not exists in dest db Pg:$dstDbName\n";
        next;
      }
	print "Clearing data from table $table in $dstDbHost\n";
    clear_table($dstDb,$table);
    next if $table=~/^cache/;
    next if $table=~/^locales/;
	print "Copying data from mysql table $table in $srcDbHost to postgres table $table in $dstDbHost\n";
    my $cnt = copy_table($srcDb,$dstDb,$table) unless $table=~/^cache/;
	$totalRowCount = $totalRowCount + $cnt;
    print "$cnt rows copied ($table)\n";
    init_seq($dstDb,$table,$seqmap->{$table}); # if (exists $seqmap->{$table});
  }  
print "Commiting changes \n";     
$dstDb->commit;
print "$totalRowCount rows were copied.\n";
exit 0;
 
 
sub init_seq  {  
    my ($db,$table,$list)=@_;
    return unless $list;
    for my $item (@{$list}) {
        my $ref = $db->selectall_arrayref("SELECT MAX($item->{field}) FROM $table") or die;
        my $val = $ref->[0]->[0];
        if ($val) {
            $db->do("SELECT SETVAL(\'$item->{seq}\',$val)") or die;
          }
      }
}  
 
sub clear_table {  
    my ($dbh,$table)=@_;
    return unless $table;
    $dbh->do("DELETE FROM $table") or die;
}  
 
sub copy_table { 
    my ($srcDb,$dstDb,$table)=@_;
    die unless $table;
    my $slf = $srcDb->prepare("select * from $table limit 1");
    my $rows = $slf->execute() or die;
    return 0 if $rows < 1;
    my $rec1 = $slf->fetchrow_hashref;
    $slf->finish;
    my @fieldnames = sort keys %$rec1;
    my @qm =  map { '?'} @fieldnames;
    my $ins = $dstDb->prepare("INSERT INTO $table (".join(",",@fieldnames).") VALUES(".join(",",@qm).")");
    my $sel = $srcDb->prepare("SELECT * FROM $table");
    $sel->execute or die;
    my $cnt = 0;
    while (my $data = $sel->fetchrow_hashref) {
        $ins->execute(map {$data->{$_}} @fieldnames) or die;
        $cnt++;
      }
    $ins->finish;
    $sel->finish;
    return $cnt;
}

Automated Capture of Webpages (Main Process)

Ivan Villareal — Fri, 25 Sep 2009 00:21:45 +0000

This post is the continuation of Xvfb and Firefox headless screenshot generator were I explained how Xvfb and Firefox can be used to get a screenshot of any web resource supported by firefox.

The next challenge I had, involved the creation of an automated process that could be called from the web or from a cron job, this process job would take a list of sites that need to be captured, and it will save its shots with their respective thumbnail somewhere, also this same process had to communicate somehow with the front end, to inform the user about the status.

So I’ve planned the architecture for this solution in a modular way

A daemon checking if new jobs were sent.
The main process that takes some parameters including the site list and save the generated screenshots somewhere.
A logger facility to communicate with an external process.
A web application to take the site list and show the results.

This is the architecture diagram:

Screen Grabber System Arquitechture

From the above modules, I will describe briefly the Main process, this main process started as a simple bash script but this wasn’t sufficient for scalability and security standpoints, so I’ve decided to make it in perl.

The current version has changed a lot, but I will describe the main aspects of this script:

The following CPAN extensions were used in this module:

This process receives the following arguments:

The path to a file where the list of urls are, the filename represents the job name and this file must contain 1 url per line (Required)
The number of seconds that firefox will have to render each url (Optional)

After the main process is called, either by a daemon or by a cron job, it will connect to a database and search the job based on the received parameter, this job needs to be inserted with an API I did in php, this API is called from the frontend and the cron jobs as well and it serves as a glue point between everything.

After the update is done, I prepare the environment for Xvfb and spawn a new Virtual Screen with its corresponding firefox instance:

This is the code used:

sub initialize {
    logThis("======= Process Starting ==========", 1);
 
    my $numArgs = $#ARGV + 1;
    if ($numArgs < 1) {
        logThis("Bad number of parameters", 1);
        die "Bad Number of params \n";
    }
 
    chomp($siteList = $ARGV[0]);
 
    if ($numArgs > 2) {
        chomp($renderTime = $ARGV[2]);
    }
    ($jobName, my $dirName, my $fileExtension) = fileparse($siteList, ('\.processing') );
    logThis("Parameters received \nSiteList: $siteList\nRelated Domain:$relatedDomain\nRender Time: $renderTime",1);
    logThis("Initializing Db",3);
    $dsn = "DBI:mysql:database=$db{'name'};host=$db{'host'};port=$db{'port'}";
    $dbh = DBI->connect($dsn, $db{'user'}, $db{'pass'}) or die('failed to connect');
    my $jobRow = my $domain = $dbh->selectrow_hashref("SELECT idJobs FROM jobs WHERE jobName = ?",undef,$jobName);
    $jobId = $jobRow->{'idJobs'};
    my $result = $dbh->do("UPDATE jobs SET dateStarted = CURRENT_TIMESTAMP WHERE jobName = ?",undef,$jobName);
 
    logThis("Conected to Db $db{'name'}",1);
    jobUpdate(2, "Starting capture process...");
    $ENV{'DISPLAY'} =':1';
    logThis("Starting virtual screen...",3);
    $pid{'xvfb'} = spawn $xvfbBin, ':1', '-screen', '0', '1024x768x24' or die "Xvfb failed to start $!";
    logThis("Xvfb Pid: $pid{'xvfb'} ",3);
    $pid{'ratpoison'} = spawn 'ratpoison' or die "spawn ratpoison failed";
    logThis("Ratpoison Pid: $pid{'ratpoison'} ",3);
    sleep(1);
    $pid{'firefox'} = spawn $ffBin, '-width 1024', '-height 768' or die "spawn $!";
    logThis("Firefox Process Id: $pid{'firefox'}",1);
    sleep(5);
    logThis("---Initialize sub end here---",4);
    return 1;
}

After the initialization I proceed to read the list of files to start capturing each one, this is the code that does exactly that:

while () {
 my $site    = $_;
 chop($site);
 ##
 # Validate that the line starts with http or https
 ##
 if ($site =~ m!\b(https?)://!i) {
    logThis("Processing: $site",4);
    captureSite($site, $relatedDomainId);
    logThis("Sites Processed" . $stats{'processed'} ."=". $stats{'succesfull'} ."+". $stats{'failed'} ."+". $stats{'unrendered'} ."entre ".$stats{'siteCount'},1);
    $stats{'percentage'} = ($stats{'processed'}/$stats{'siteCount'})*100;
    updateStatus('stats', \%stats);
 }
 }file>

And the captureSite sub is where the magic is, here it is:

sub captureSite {
    my $site          = shift;
    my $result;
    my $imgName;
    my $imageStatus = '';
    my $fileNameExit = 0;
    my $imgCounter = 0;
    my $imgFullName;
    my $thumbFile;
 
    updateStatus('status', "Processing $site");
    jobUpdate(2, "Processing $site");
    logThis("Starting to capture: $site",2);
##
# Check the domain for valid parts
##
    my @domainParts = $site =~ m/\b((?#protocol)https?|ftp):\/\/((?#domain)[-A-Z0-9.]+)((?#file)\/[-A-Z0-9+&@#\/%=~_|!:,.;]*)?((?#parameters)\?[-A-Z0-9+&@#\/%=~_|!:,.;]*)?/ig;
 
    if ($domainParts[0] && $domainParts[1]) {
        if ($openedTabs > $maximumTabs) {
            logThis("Maximum Number of Opened Tabs Reached Killing Firefox pid $pid{'firefox'}",3);
            kill 0, $pid{'firefox'};
            sleep(2);
            $pid{'firefox'} = spawn $ffBin, '-width 1024', '-height 768' or die "spawn $!";
            logThis("New Firefox Process Id: $pid{'firefox'}",1);
            sleep(3);
            $openedTabs = 1;
        }
#because ff is tabless the old page doesn't disappear until
#all content is loaded, so I should remove content first.
        $cmd = $ffBin . ' -remote "openUrl(http://localhost/blank.html)"';
        my $res = `$cmd`;
        sleep(1);
        $cmd = $ffBin . ' -remote "openUrl('.$site.')"';
        logThis("Opening $site",1);
        logThis($cmd, 3);
        $res = `$cmd`;
        $openedTabs++;
        logThis("Waiting $renderTime seconds to allow page rendering",3);
        sleep($renderTime);
##
# We remove harmfull charachters
# maybe an md5sum would be better, because
# we would always have the same number of chars
##
        $imgName = $site;
        $imgName =~ s!\b(https?)://|(www\.)!!g; #removes http://www
            $imgName =~ s!\/|\#|\?$!!g; #removes \ # ? from the end
            $imgName =~ s![^0-9^A-Z^a-z^_^.]!_!g;
#cut excessive large filenames
        if (length($imgName) > $maxFileNameLength) {
            $imgName = substr $imgName, 0, $maxFileNameLength;
        }
        my $baseImgName = $imgName;
        while (!$fileNameExit) {
            $imgFullName = $saveDir . $imgName . $saveFormat;
            $thumbFile   = $thumbsDir . 'thumb_' . $imgName . $saveFormat;
            logThis($imgFullName,1);
            logThis("Cecking if image already exists",1);
            if (-e $imgFullName) {
                $imgCounter++;
                $imgName = $baseImgName . '_' . $imgCounter;
            } else {
                $fileNameExit = 1;
            }
        }
#creating the screenshot
        $cmd = 'import -window root ' . $imgFullName;
        logThis("Command: " .$cmd, 3);
        $res   = `$cmd`;
        if (-e $imgFullName) {
            logThis("The image $imgFullName was created succesfully",1);
            $imageStatus = 'success';
            my $fileSize = -s $imgFullName;
            logThis("File Size for image created: $fileSize",1);
# Check the filesize, if its to low wait give more time to render the page
            if ($fileSize < = $minFileSize) {
                logThis("File is too small, I think the page has not finished rendering, giving it more time...",2);
                my $exit = 0;
                my $try = 0;
                while (!$exit) {
                    logThis("Giving the page $renderTime more seconds to render...",2);
                    updateStatus('status', "Giving $site more time to render...");
                    sleep($renderTime);
                    logThis("Command: " .$cmd, 3);
                    $res   = `$cmd`;
                    $fileSize = -s $imgFullName;
                    logThis("File Size for image created: $fileSize",1);
                    if ($fileSize > $minFileSize) {
                        $exit = 1;
                        $imageStatus = 'success';
                    } else {
                        $try++;
                    }
                    if ($try > 3 ) {
                        $exit = 1;
                        $imageStatus = 'failed';
                    }
                }
            }
            logThis("Croping image...",1);
            $cmd = "convert $imgFullName -crop '1024x768+0+24' $imgFullName";
            $res = `$cmd`;
 
            logThis("Creating Thumbnail in $thumbFile",1);
            $cmd = "convert $imgFullName -resize '170x128' $thumbFile";
            $res = `$cmd`;
            my %image = (
                    site   => $site,
                    image  => $imgFullName,
                    thumb  => $thumbFile,
                    status => $imageStatus,
                    );
            push(@images, \%image);
            insertImage($site, $imgFullName, $thumbFile, $relatedDomainId, $imageStatus);
            if ($imageStatus eq 'success') {
                $stats{'succesfull'}++;
            } else {
                $stats{'failed'}++;
            }
        } else {
            logThis("Fail creating the image",1);
            $stats{'unrendered'}++;
        }
    } else {
        $stats{'unrendered'}++;
    }
    $stats{'processed'}++;
    jobUpdate(2, "Finished processing $site");
}

As you can see I’m updating the job status in the jobUpdate sub, also there are other subs to take care of inserting the screenshot info in the db.

When this program ends, it updates the db to let the inform the daemon that is ready to take more jobs.

In the next post I will be providing info on how the frontend works, and what the final results of this implementation ended.