Create a backup with mysqldump and compress the stream with bzip2

ivan@mini:~$ mysqldump –h dbHost –u dbUser –pdbPass –add-drop-database –databases <dbname> | bzip2 > dbName-02-17-2010.sql.bz2</dbname>

If I want to restore the backup I use:

ivan@mini:~$ bunzip2 < dbName-02-17-2010.sql.bz2 | mysql –h dbHost –u dbUser –pdbPass

I can switch bzip2 compression by changing only the program name for example gzip/gunzip lzma/unlzma

This servers didn’t came with a control panel (better for me), so the first thing I did was to login:

ivan@mini:~$ ssh root@23.45.12.56

After this I checked the OS, version and architecture

[root@V100205C4HB9V-1 ~]# cat /etc/*release*
CentOS release 5.4 (Final)
[root@V100205C4HB9V-1 ~]# uname -a
Linux V100205C4HB9V-1 2.6.18-028stab064.7 #1 SMP Wed Aug 26 13:11:07 MSD 2009 x86_64 x86_64 x86_64 GNU/Linux
[root@V100205C4HB9V-1 ~]#

Ok, now that I know the OS I create a normal user to avoid using the root account.

[root@V100205C4HB9V-1 ~]# useradd –Gwheel ivan

Then I change the server name to mygdon

[root@V100205C4HB9V-1 /]# sed -i 's/V100205C4HB9V-1/mygdon/g' /etc/sysconfig/network
[root@V100205C4HB9V-1 ~]# sed -i 's/V100205C4HB9V-1/mygdon/g' /etc/hosts
[root@V100205C4HB9V-1 ~]# echo HOST.DOMAIN.com > /etc/hostname
[root@V100205C4HB9V-1 ~]# hostname -F /etc/hostname

Unfortunately I was unable to persist the new hostname, because it is a VPS server, there are ways around this, but didn’t have the time to make the changes so I just moved on, and leave this for later

The next thing I did was update the OS, and add the rpmforge repo because I will need some apps from there

[root@V100205C4HB9V-1 ~]# yum update
[root@V100205C4HB9V-1 ~]# yum upgrade
[root@V100205C4HB9V-1 ~]# wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm
[root@V100205C4HB9V-1 ~]# rpm -Uhv rpmforge-release-0.5.1-1.$dist.rf.$arch.rpm
[root@V100205C4HB9V-1 ~]# yum update
[root@V100205C4HB9V-1 ~]# yum upgrade

Then I installed some packages I often use:

[root@V100205C4HB9V-1 ~]# yum install htop screen vim-enhanced

The VPS already had a web stack installed I just did some configuration:

Configuring Apache Virtual Hosts

[root@V100205C4HB9V-1 ~]# mkdir /var/www/vhosts/{site1, site2} –p
[root@V100205C4HB9V-1 ~]# vi /etc/httpd/conf/httpd.conf

Here I Uncommented the following directive

NameVirtualHost *:80

and Added a default vhost a new vhost

<virtualhost *:80>
    DocumentRoot /var/www/vhosts/default
    ServerName mygdon.site1.net
    <directory /var/www/vhosts/default>
        Options Indexes FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
    </directory>
    ErrorLog logs/mygdon.site1.net-error_log
    CustomLog logs/mygdon.site1.net-access_log common
</virtualhost>
 
<virtualhost *:80>
    DocumentRoot /var/www/vhosts/site2
    ServerName appname.site2.net
    <directory /var/www/vhosts/site2>
        Options Indexes FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
    </directory>
    ErrorLog logs/appname.site2.net-error_log
    CustomLog logs/appname.site2.net-access_log common
</virtualhost>

then restarted the apache server:

[root@V100205C4HB9V-1 ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting  httpd:                                            [  OK  ]

and  I wrote a simple php file to test this out:

[root@V100205C4HB9V-1 ~]# echo '<? phpinfo(); ?>' > /var/www/vhosts/site1/index.php

And because I haven’t configured a DNS server I just temporarily added the server to my hosts file (On my local machine):

root@mini:/etc# echo 'subdomain.site1.net 54.65.74.23' > /etc/hosts

and here is the result:

Configuring Mysql

Once I had the virtual hosts configuration in place I added a mysql user and changed the default root password:

[root@V100205C4HB9V-1 vhosts]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.0.77 Source distribution
 
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
 
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| test               |
+--------------------+
3 rows in set (0.00 sec)
 
mysql> UPDATE mysql.user SET Password=PASSWORD('pass-here') WHERE user='root';
Query OK, 3 rows affected (0.02 sec)
Rows matched: 3  Changed: 3  Warnings: 0
 
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)
 
mysql> CREATE USER 'ivan'@'localhost' IDENTIFIED BY 'pass-here';
Query OK, 0 rows affected (0.01 sec)
 
mysql> GRANT ALL PRIVILEGES ON *.* TO 'ivan'@'localhost' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)
 
mysql> CREATE USER 'ivan'@'%' IDENTIFIED BY 'pass-here';
Query OK, 0 rows affected (0.00 sec)
 
mysql> GRANT ALL PRIVILEGES ON *.* TO 'ivan'@'%' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)
 
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
 
mysql>

Now the server is  ready for adding some virtual hosts, I just need to configure a DNS server and the mail server, but because the DNS is on another server and already working I just have to add a new zone, as for mail server, this server will be only used to send emails so I don’t need to dig in postfix configuration files for now.

I’ll write another post regarding the security, but for now this will work.

Anyway, right now I don’t have time to spend getting a new OS on this machine, I just want something that works, I don’t plan using this to work, but I would like to have some of the tools I use into this.

So my options were, trying Ubuntu remix or Windows Xp, after checking that hp had the drivers for XP, I was going to change Windows 7 to Xp, but I was reading how to create a USB boot drive, and it was too much hassle for me. (I’m very lazy for any Microsoft stuff).

My next option was Ubuntu remix, I’m not a big fan of Ubuntu, but I wanted something to just worked out of the box, also this little netbook comes with splashtop, a really nice feature for checking stuff online fast, and I didn’t want to loose this so I really didn’t knew how well this was going to work.

I started downloading Ubuntu 9.10 Karmic yesterday night, just in case, Today after more frustrating things, I decided to give Karmic a try.

I grabbed my usb pendrive, a 1gb  kingston traveler, mounted the iso and ran usbinst.exe, it was pretty straightforward, my only advice to you would be to avoid saving space for your documents in the usb, my first try I set this to 30Mb and the boot failed, so I set the do not save feature.

It took about 40 minutes to finish copying the files, after it was done, I restarted the netbook with the pendrive connected, and at the first boot screen I pressed F9 key for booting options, selected the pendrive and that was it.

After testing ubuntu from the pendrive, it ran pretty well and everything but the wifi worked, I did an installation, keeping the windows partition and the hp restore partition, just in case.

The installation was pretty smooth, no issues at all, after the first boot I just installed the broadcom drivers, it was pretty easy with the driver manager in ubuntu, and the wifi was working

Update: My hp mini was stolen by a couple of elderly people, I saw the video , so I’m unable to play more with this, but the time I had this I was very happy with it, it performed well.

This works well when I need to make a quick change or test 1 file, but If I have to synchronize several files it is a very time consuming task.

So what I use in this cases is sshfs it is a pretty cool tool, to allow me to mount any filesystem so I can use any rsync on a remote server without dealing with opening ports configure, the server or do other stuff,

Gentoo is my main OS, so it was really easy to have this working, I had to reconfigure the kernel to support fuse

Symbol: FUSE_FS [=m]
Location:
  -> File systems
      -> Filesystem in Userspace support

and after this was ready I only installed sys-fs/sshfs-fuse

[I] sys-fs/sshfs-fuse
     Available versions:  1.9 ~2.1 2.2
     Installed versions:  2.2(12:33:09 PM 09/08/2009)
     Homepage:            http://fuse.sourceforge.net/sshfs.html
     Description:         Fuse-filesystem utilizing the sftp service.

when it was ready I did the following to mount the remote filesystem:

ivan@gondor ~ $ sshfs phpfix@prestant.citizenhawk.net: preasent/

After this I’m able to use the remote file system as usual.

First check what is the version of the distro:

cat /etc/*release*

Then make an update:

yum update

After 496 package updates I’ve installed php5

yum --enablerepo=centosplus install php

This were the installed packages:

php-5.1.6-3.el4s1.10
php-cli-5.1.6-3.el4s1.10
php-common-5.1.6-3.el4s1.10

So I’ve created a info.php file, and restarted the apache server:

echo "<?php phpinfo(); ?>" > /var/www/html/info.php && service httpd restart

It was working correctly, so the next thing was to have mysql 5 in it, so I did:

yum --enablerepo=centosplus install mysql-server

I got some problems here, first I got this:

Transaction Check Error: file /etc/my.cnf from install of mysql-libs-5.0.68-1.el4_6 conflicts with file from package mysql-4.1.22-2.el4
file /usr/share/mysql/charsets/Index.xml from install of mysql-libs-5.0.68-1.el4_6 conflicts with file from package mysql-4.1.22-2.el4
file /usr/share/mysql/charsets/README from install of mysql-libs-5.0.68-1.el4_6 conflicts with file from package mysql-4.1.22-2.el4
file /usr/share/mysql/charsets/armscii8.xml from install of mysql-libs-5.0.68-1.el4_6 conflicts with file from package mysql-4.1.22-2.el4
file /usr/share/mysql/charsets/ascii.xml from install of mysql-libs-5.0.68-1.el4_6 conflicts with file from package mysql-4.1.22-2.el4

So what I found was that yum was trying to install mysql.i386 but the architechture is x86_64, to fix this I had to specify the arch

yum --enablerepo=centosplus install mysql.x86_64 mysql-server

After this I was able to install it, but when I tried to start the server I received some errors about error messages not right, I’ve checked the installed packages and it was installed mysqlclient10, so I removed the error messages problem disappeared, however I still couldn’t startup the server, checking at the logs I found this:

InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
091012 11:01:18  InnoDB: Started; log sequence number 0 0                       091012 11:01:18 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist                                                     091012 11:01:18  mysqld ended                                               

To fix this I cleared the data dir, that had som files, this was in ‘/var/lib/mysql’ and after that I ran mysql_install_db:

[root@prestant /]# cd /var/lib/mysql 
[root@prestant mysql]# rm -rf *
[root@prestant mysql]# mysql_install_db --user=mysql -ldata=/var/lib/mysql
[root@prestant mysql]# service mysqld start

After the server was up, I changed the root password and added a new user:

[root@prestant mysql]# mysqladmin -u root password NEWPASSWORD
[root@prestant mysql]# mysql -pNEWPASSWORD
 

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.0.82sp1 Source distribution
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
mysql> CREATE USER 'ivan'@'%' IDENTIFIED BY 'passhere';
Query OK, 0 rows affected (0.00 sec)
 
mysql> GRANT ALL PRIVILEGES ON *.* TO 'ivan'@'%' IDENTIFIED BY 'passhere' WITH GRANT OPTION MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0 ;
Query OK, 0 rows affected (0.00 sec)
 
mysql> CREATE USER 'ivan'@'localhost' IDENTIFIED BY 'passhere';
Query OK, 0 rows affected (0.00 sec)
 
mysql> GRANT ALL PRIVILEGES ON *.* TO 'ivan'@'localhost' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)
 
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
 
mysql> exit
Bye
 
 

About 2 years ago I purchased this 500Gb Acomdata drive that was on sale at Fry’s, it was a good deal back then, cost me about 100 bucks, it had an aluminium case and a power off switch, and I was in the need to expand the storage capacity of my multimedia server, which is linux based.

I didn’t researched this product, I didn’t even knew that acomdata was a brand, anyway I remember seeing the “Hybrid Drive” statement, but didn’t care, I was going to wipe this drive an reformat it as ext3, so I was aware that all the “software features” it had were useless to me, like the PushButton™ Backup and the Nomad Mobile Desktop.

After I arrived home, I plugged this thing on my server, which had ivman and a custom script to mount anything, but after few seconds nothing happened, I look at the kernel messages and I started to worry, It showed as a CD drive instead of a Mass Storage Device, I tried unsuccessfully to mount this manually, so I thought this drive must be bad.

The next day I connected this drive to a Windows XP, and after some drivers install, it was working, I noticed that it created a virtual CD drive on Windows it had some software on it, so I thought that this was some kind of partition that would be erased after I formatted the drive, so I did this, but the partition was still there.

I did some research on the internet but didn’t found anything use full, so I was again trying to make this stupid drive worked on linux, I hate when I have to fight with the stuff I buy just to make it work on linux, I was very annoyed by this, finally after looking the kernel messages I’ve found that the drive was acting as a SCSI device, so I activated “Probe all LUNs on each SCSI device” in the kernel and after a kernel restart both partitions were available on Linux, however this still had a problem, because I was using ivman to detect the drive and mounted, ivman  didn’t worked for hard drive partition, just for the virtual CD, I’ve tried unsuccessfully to delete that stupid CD partition on windows and Linux.

I had enough of this and I was going to take this drive back to the store, but in a desperation moment I took my screwdriver and started to tearing apart the drive, I wasn’t even stopped by the warranty void seal, my intention was to take the drive format it and put it back without the cd part, I did this, and the format was successful, the drive was totally empty, but when I put this back to the enclosure the stupid CD partition was there, empty this time, but it was there, so it was a firmware issue and I didn’t found a firmware upgrade to remove this stupid feature.

I did some scripts to detect this unmounted the CD part and mount the hard drive partition, it took me about a week to have this thing working the way I wanted,  this was  unbelievable, why they don’t put the software to remove this “hybryd feature”.

However this drive has been working fine, with the virtual CD partition still there until Yesterday, I was in the need to transfer some large files from work to home, and this was the drive more reachable from all the drives I have, so I bring it her to the office, and as soon as I plugged in the CD partition appeared on my Desktop, so I thought it has been a while now, lets see if I can finally reflash the firmware to remove this so called feature, and I’ve found a forum with several users trying to remove this, and suddenly I’ve found this  post:

You need this software from http://www.mediafire.com/?bgwuwq5xzbm

unzip and click on MP251MFG, click on “configure hdd” and once it says pass just unplug and plug in the device. CD PART gone. 

I quickly downloaded the file in question, it was a zip file called “REMOVE CD PART.zip” with about 1.5Mb in size, it has  a Readme.pdf that describes the MP251MFG program, I booted up in Windows, installed the fsdriver because this drive was an ext3, plugged the drive and run the MP251MFG.exe application this is what I saw:

I didn’t backed up the drive, I was just exploring the options and I clicked the Configure HDD, a process was started and finished successfully, I was affraid that my data were gone, so I headed to check if the data was there, and it was, I copied the important stuff somewhere else, unplugged the drive and plugged back in, and voilà, the CD partition was gone along with the data in the drive, but that din’t matter the CD partition was really gone

The drive was reformatted to FAT32, so I reformatted back to ext3, and now it is working pretty well, 2 years later.

I’ve found that this program works for other drives too, here are some links where this program can be found

For Windows:

• xSil251 Switch HDD to CD+HDD_Nomad Mobile Desktop_V1.04_091007
• xSil251 Switch CD+HDD to HDD 3.5_Toshiba_V1.02_041307 

For Macintosh:

• Switch to CD+Secure Nomad for Mac
• CD to HDD Convertor for Mac

However I’m constantly in the need to login to some servers just to make a quick check of something, so I have to type in my credentials every time I log in, this really annoys me, because when I’m focused on something I don’t want to loose that focus waiting for the login prompt.

So what I did  to fix this annoyance, was to use Key Authentication  instead of Password Authentication, I’ve been using this for several years now under several distros without a problem.

So in short this is what I do, whenever I want to use Key Auth.

gondor .ssh # cd ~/.ssh
gondor .ssh ~ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
53:df:bb:74:24:2d:7f:78:1f:85:66:f0:19:b6:9f:71 root@gondor
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|          . . o  |
|         . . = * |
|        S   . @.E|
|         .   o O=|
|              +o*|
|             . ++|
|              . .|
+-----------------+

After my private and public keys are generated I copy the public key to the server where I want to access without password

gondor .ssh # ssh-copy-id -i id_rsa.pub ivan@odin
The authenticity of host 'odin (192.168.1.201)' can't be established.
RSA key fingerprint is bb:36:b4:0b:05:13:ce:a2:2e:95:97:59:65:f3:f8:a8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'odin,192.168.1.201' (RSA) to the list of known hosts.
Password:
Now try logging into the machine, with "ssh 'ivan@odin'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

gondor .ssh # ssh ivan@odin
Last login: Fri Sep 25 15:50:03 PDT 2009 from gondor.artedigital on ssh
Last login: Fri Sep 25 15:50:07 2009 from gondor.artedigital
ivan@odin ~ $

The next challenge I had, involved the creation of an automated process that could be called from the web or from a cron job, this process job would take a list of sites that need to be captured, and it will save its shots with their respective thumbnail somewhere, also this same process had to communicate somehow with the front end, to inform the user about the status.

So I’ve planned the architecture for this solution in a modular way

• A daemon checking if new jobs were sent.
• The main process that takes some parameters including the site list and save the generated screenshots somewhere.
• A logger facility to communicate with an external process.
• A web application to take the site list and show the results.

This is the architecture diagram:

Screen Grabber System Arquitechture

From the above modules, I will describe briefly the Main process, this main process started as a simple bash script but this wasn’t sufficient for scalability and security standpoints, so I’ve decided to make it in perl.

The current version has changed a lot, but I will describe the main aspects of this script:

The following CPAN extensions were used in this module:

• Run
• DBI
• File::Basename
• Serialize

This process receives the following arguments:

1. The path to a file where the list of urls are, the filename represents the job name and this file must contain 1 url per line (Required)
2. The number of seconds that firefox will have to render each url (Optional)

After the main process is called, either by a daemon or by a cron job, it will connect to a database and search the job based on the received parameter, this job needs to be inserted with an API I did in php, this API is called from the frontend and the cron jobs as well and it serves as a glue point between everything.

After the update is done, I prepare the environment for Xvfb and spawn a new Virtual Screen with its corresponding firefox instance:

This is the code used:

sub initialize {
    logThis("======= Process Starting ==========", 1);
 
    my $numArgs = $#ARGV + 1;
    if ($numArgs < 1) {
        logThis("Bad number of parameters", 1);
        die "Bad Number of params \n";
    }
 
    chomp($siteList = $ARGV[0]);
 
    if ($numArgs > 2) {
        chomp($renderTime = $ARGV[2]);
    }
    ($jobName, my $dirName, my $fileExtension) = fileparse($siteList, ('\.processing') );
    logThis("Parameters received \nSiteList: $siteList\nRelated Domain:$relatedDomain\nRender Time: $renderTime",1);
    logThis("Initializing Db",3);
    $dsn = "DBI:mysql:database=$db{'name'};host=$db{'host'};port=$db{'port'}";
    $dbh = DBI->connect($dsn, $db{'user'}, $db{'pass'}) or die('failed to connect');
    my $jobRow = my $domain = $dbh->selectrow_hashref("SELECT idJobs FROM jobs WHERE jobName = ?",undef,$jobName);
    $jobId = $jobRow->{'idJobs'};
    my $result = $dbh->do("UPDATE jobs SET dateStarted = CURRENT_TIMESTAMP WHERE jobName = ?",undef,$jobName);
 
    logThis("Conected to Db $db{'name'}",1);
    jobUpdate(2, "Starting capture process...");
    $ENV{'DISPLAY'} =':1';
    logThis("Starting virtual screen...",3);
    $pid{'xvfb'} = spawn $xvfbBin, ':1', '-screen', '0', '1024x768x24' or die "Xvfb failed to start $!";
    logThis("Xvfb Pid: $pid{'xvfb'} ",3);
    $pid{'ratpoison'} = spawn 'ratpoison' or die "spawn ratpoison failed";
    logThis("Ratpoison Pid: $pid{'ratpoison'} ",3);
    sleep(1);
    $pid{'firefox'} = spawn $ffBin, '-width 1024', '-height 768' or die "spawn $!";
    logThis("Firefox Process Id: $pid{'firefox'}",1);
    sleep(5);
    logThis("---Initialize sub end here---",4);
    return 1;
}

After the initialization I proceed to read the list of files to start capturing each one, this is the code that does exactly that:

 
while (<FILE>) {
 my $site    = $_;
 chop($site);
 ##
 # Validate that the line starts with http or https
 ##
 if ($site =~ m!\b(https?)://!i) {
    logThis("Processing: $site",4);
    captureSite($site, $relatedDomainId);
    logThis("Sites Processed" . $stats{'processed'} ."=". $stats{'succesfull'} ."+". $stats{'failed'} ."+". $stats{'unrendered'} ."entre ".$stats{'siteCount'},1);
    $stats{'percentage'} = ($stats{'processed'}/$stats{'siteCount'})*100;
    updateStatus('stats', \%stats);
 }
 }

And the captureSite sub is where the magic is, here it is:

sub captureSite {
    my $site          = shift;
    my $result;
    my $imgName;
    my $imageStatus = '';
    my $fileNameExit = 0;
    my $imgCounter = 0;
    my $imgFullName;
    my $thumbFile;
 
    updateStatus('status', "Processing $site");
    jobUpdate(2, "Processing $site");
    logThis("Starting to capture: $site",2);
##
# Check the domain for valid parts
##
    my @domainParts = $site =~ m/\b((?#protocol)https?|ftp):\/\/((?#domain)[-A-Z0-9.]+)((?#file)\/[-A-Z0-9+&@#\/%=~_|!:,.;]*)?((?#parameters)\?[-A-Z0-9+&@#\/%=~_|!:,.;]*)?/ig;
 
    if ($domainParts[0] && $domainParts[1]) {
        if ($openedTabs > $maximumTabs) {
            logThis("Maximum Number of Opened Tabs Reached Killing Firefox pid $pid{'firefox'}",3);
            kill 0, $pid{'firefox'};
            sleep(2);
            $pid{'firefox'} = spawn $ffBin, '-width 1024', '-height 768' or die "spawn $!";
            logThis("New Firefox Process Id: $pid{'firefox'}",1);
            sleep(3);
            $openedTabs = 1;
        }
#because ff is tabless the old page doesn't disappear until
#all content is loaded, so I should remove content first.
        $cmd = $ffBin . ' -remote "openUrl(http://localhost/blank.html)"';
        my $res = `$cmd`;
        sleep(1);
        $cmd = $ffBin . ' -remote "openUrl('.$site.')"';
        logThis("Opening $site",1);
        logThis($cmd, 3);
        $res = `$cmd`;
        $openedTabs++;
        logThis("Waiting $renderTime seconds to allow page rendering",3);
        sleep($renderTime);
##
# We remove harmfull charachters
# maybe an md5sum would be better, because
# we would always have the same number of chars
##
        $imgName = $site;
        $imgName =~ s!\b(https?)://|(www\.)!!g; #removes http://www
            $imgName =~ s!\/|\#|\?$!!g; #removes \ # ? from the end
            $imgName =~ s![^0-9^A-Z^a-z^_^.]!_!g;
#cut excessive large filenames
        if (length($imgName) > $maxFileNameLength) {
            $imgName = substr $imgName, 0, $maxFileNameLength;
        }
        my $baseImgName = $imgName;
        while (!$fileNameExit) {
            $imgFullName = $saveDir . $imgName . $saveFormat;
            $thumbFile   = $thumbsDir . 'thumb_' . $imgName . $saveFormat;
            logThis($imgFullName,1);
            logThis("Cecking if image already exists",1);
            if (-e $imgFullName) {
                $imgCounter++;
                $imgName = $baseImgName . '_' . $imgCounter;
            } else {
                $fileNameExit = 1;
            }
        }
#creating the screenshot
        $cmd = 'import -window root ' . $imgFullName;
        logThis("Command: " .$cmd, 3);
        $res   = `$cmd`;
        if (-e $imgFullName) {
            logThis("The image $imgFullName was created succesfully",1);
            $imageStatus = 'success';
            my $fileSize = -s $imgFullName;
            logThis("File Size for image created: $fileSize",1);
# Check the filesize, if its to low wait give more time to render the page
            if ($fileSize <= $minFileSize) {
                logThis("File is too small, I think the page has not finished rendering, giving it more time...",2);
                my $exit = 0;
                my $try = 0;
                while (!$exit) {
                    logThis("Giving the page $renderTime more seconds to render...",2);
                    updateStatus('status', "Giving $site more time to render...");
                    sleep($renderTime);
                    logThis("Command: " .$cmd, 3);
                    $res   = `$cmd`;
                    $fileSize = -s $imgFullName;
                    logThis("File Size for image created: $fileSize",1);
                    if ($fileSize > $minFileSize) {
                        $exit = 1;
                        $imageStatus = 'success';
                    } else {
                        $try++;
                    }
                    if ($try > 3 ) {
                        $exit = 1;
                        $imageStatus = 'failed';
                    }
                }
            }
            logThis("Croping image...",1);
            $cmd = "convert $imgFullName -crop '1024x768+0+24' $imgFullName";
            $res = `$cmd`;
 
            logThis("Creating Thumbnail in $thumbFile",1);
            $cmd = "convert $imgFullName -resize '170x128' $thumbFile";
            $res = `$cmd`;
            my %image = (
                    site   => $site,
                    image  => $imgFullName,
                    thumb  => $thumbFile,
                    status => $imageStatus,
                    );
            push(@images, \%image);
            insertImage($site, $imgFullName, $thumbFile, $relatedDomainId, $imageStatus);
            if ($imageStatus eq 'success') {
                $stats{'succesfull'}++;
            } else {
                $stats{'failed'}++;
            }
        } else {
            logThis("Fail creating the image",1);
            $stats{'unrendered'}++;
        }
    } else {
        $stats{'unrendered'}++;
    }
    $stats{'processed'}++;
    jobUpdate(2, "Finished processing $site");
}

As you can see I’m updating the job status in the jobUpdate sub, also there are other subs to take care of  inserting the screenshot info in the db.

When this program ends, it updates the db to let the inform the daemon that is ready to take more jobs.

In the next post I will be providing  info on how the frontend works, and what the final results of this implementation ended.

• It has to be fast
• It needs to run on linux

I did some research, trying to found the best approach, I already knew Xvfb but haven’t worked with it in a real project, there wasn’t much info about what I was trying to do, however I’ve found some other automated solutions made in .NET that worked on windows using IE as rendering engine, but I’ve found nothing that worked out of the box in linux

I did found some interesting setups but nothing similar to what I was trying to achieve, and all the directions I’ve got, pointed me to the same place Xvfb, so because I’m a hands-on guy, I quickly cloned a minimal debian box to start playing with this.

First I’ve installed Xvfb, firefox and some fonts

ivano ~ # apt-get install xvfb xfonts-base xfonts-75dpi xfonts-100dpi firefox

After some package downloads I’ve got everything installed, I’ve checked the man page and found:

-screen screennum WxHxD

This option creates screen screennum and sets its width, height,

and depth to W, H, and D respectively. By default, only screen 0

exists and has the dimensions 1280×1024x12.

So I did:

ivano ~ # Xvfb :1 -screen 0 1024x768x24 &
[1] 2665
ivano ~ # Could not init font path element /usr/X11R6/lib/X11/fonts/misc, removing from list!
Could not init font path element /usr/share/fonts/X11/cyrillic, removing from list!
Could not init font path element /usr/share/fonts/X11/Type1, removing from list!
Could not init font path element /usr/X11R6/lib/X11/fonts/Type1, removing from list!
Could not init font path element /var/lib/defoma/x-ttcidfont-conf.d/dirs/TrueType, removing from list!

I’ve got some warnings from X, but nothing lethal, so now I have a Virtual X server ready to throw anything at it.

ivano ~ # DISPLAY=:1 firefox http://google.com &

I have to set the DISPLAY variable to be the same as the virtual screen I’ve created, after this command firefox must be there so I need to get a screen shot of the virtual screen so I’ve used the image magick import tool

ivano ~ # DISPLAY=:1 import -window root google.com.png

When the import program ended I had a the following pg file:

Firefox capture within Xvfb

This apparently worked well, however I did some more testing to found any possible problem and the only thing I’ve found was that firefox tried to restore the session because of the improper shutdown so I had to tweak the firefox profile to avoid this, apart from this there were no other significant issues.

I will talk more about how I implemented the backend and frontend for the entire solution.