Ivan Villareal

About 2 years ago I purchased this 500Gb Acomdata drive that was on sale at Fry’s, it was a good deal back then, cost me about 100 bucks, it had an aluminium case and a power off switch, and I was in the need to expand the storage capacity of my multimedia server, which is linux based.

I didn’t researched this product, I didn’t even knew that acomdata was a brand, anyway I remember seeing the “Hybrid Drive” statement, but didn’t care, I was going to wipe this drive an reformat it as ext3, so I was aware that all the “software features” it had were useless to me, like the PushButton™ Backup and the Nomad Mobile Desktop.

After I arrived home, I plugged this thing on my server, which had ivman and a custom script to mount anything, but after few seconds nothing happened, I look at the kernel messages and I started to worry, It showed as a CD drive instead of a Mass Storage Device, I tried unsuccessfully to mount this manually, so I thought this drive must be bad.

The next day I connected this drive to a Windows XP, and after some drivers install, it was working, I noticed that it created a virtual CD drive on Windows it had some software on it, so I thought that this was some kind of partition that would be erased after I formatted the drive, so I did this, but the partition was still there.

I did some research on the internet but didn’t found anything use full, so I was again trying to make this stupid drive worked on linux, I hate when I have to fight with the stuff I buy just to make it work on linux, I was very annoyed by this, finally after looking the kernel messages I’ve found that the drive was acting as a SCSI device, so I activated “Probe all LUNs on each SCSI device” in the kernel and after a kernel restart both partitions were available on Linux, however this still had a problem, because I was using ivman to detect the drive and mounted, ivman  didn’t worked for hard drive partition, just for the virtual CD, I’ve tried unsuccessfully to delete that stupid CD partition on windows and Linux.

I had enough of this and I was going to take this drive back to the store, but in a desperation moment I took my screwdriver and started to tearing apart the drive, I wasn’t even stopped by the warranty void seal, my intention was to take the drive format it and put it back without the cd part, I did this, and the format was successful, the drive was totally empty, but when I put this back to the enclosure the stupid CD partition was there, empty this time, but it was there, so it was a firmware issue and I didn’t found a firmware upgrade to remove this stupid feature.

I did some scripts to detect this unmounted the CD part and mount the hard drive partition, it took me about a week to have this thing working the way I wanted,  this was  unbelievable, why they don’t put the software to remove this “hybryd feature”.

However this drive has been working fine, with the virtual CD partition still there until Yesterday, I was in the need to transfer some large files from work to home, and this was the drive more reachable from all the drives I have, so I bring it her to the office, and as soon as I plugged in the CD partition appeared on my Desktop, so I thought it has been a while now, lets see if I can finally reflash the firmware to remove this so called feature, and I’ve found a forum with several users trying to remove this, and suddenly I’ve found this  post:

You need this software from http://www.mediafire.com/?bgwuwq5xzbm

unzip and click on MP251MFG, click on “configure hdd” and once it says pass just unplug and plug in the device. CD PART gone. 

I quickly downloaded the file in question, it was a zip file called “REMOVE CD PART.zip” with about 1.5Mb in size, it has  a Readme.pdf that describes the MP251MFG program, I booted up in Windows, installed the fsdriver because this drive was an ext3, plugged the drive and run the MP251MFG.exe application this is what I saw:

I didn’t backed up the drive, I was just exploring the options and I clicked the Configure HDD, a process was started and finished successfully, I was affraid that my data were gone, so I headed to check if the data was there, and it was, I copied the important stuff somewhere else, unplugged the drive and plugged back in, and voilà, the CD partition was gone along with the data in the drive, but that din’t matter the CD partition was really gone

The drive was reformatted to FAT32, so I reformatted back to ext3, and now it is working pretty well, 2 years later.

I’ve found that this program works for other drives too, here are some links where this program can be found

For Windows:

• xSil251 Switch HDD to CD+HDD_Nomad Mobile Desktop_V1.04_091007
• xSil251 Switch CD+HDD to HDD 3.5_Toshiba_V1.02_041307 

For Macintosh:

• Switch to CD+Secure Nomad for Mac
• CD to HDD Convertor for Mac

Most part of the time I’ve spent on my development machine I have several terminals opened, but thanks to screen I only have one per server.

However I’m constantly in the need to login to some servers just to make a quick check of something, so I have to type in my credentials every time I log in, this really annoys me, because when I’m focused on something I don’t want to loose that focus waiting for the login prompt.

So what I did  to fix this annoyance, was to use Key Authentication  instead of Password Authentication, I’ve been using this for several years now under several distros without a problem.

So in short this is what I do, whenever I want to use Key Auth.

gondor .ssh # cd ~/.ssh
gondor .ssh ~ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
53:df:bb:74:24:2d:7f:78:1f:85:66:f0:19:b6:9f:71 root@gondor
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|          . . o  |
|         . . = * |
|        S   . @.E|
|         .   o O=|
|              +o*|
|             . ++|
|              . .|
+-----------------+

I haven’t set a password for the key,  because if I set a password I would have to type the password for the key each time, or setup a program to remember the key password like Gnome Keyring or Putty Pageant in Windows.

After my private and public keys are generated I copy the public key to the server where I want to access without password

gondor .ssh # ssh-copy-id -i id_rsa.pub ivan@odin
The authenticity of host 'odin (192.168.1.201)' can't be established.
RSA key fingerprint is bb:36:b4:0b:05:13:ce:a2:2e:95:97:59:65:f3:f8:a8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'odin,192.168.1.201' (RSA) to the list of known hosts.
Password:
Now try logging into the machine, with "ssh 'ivan@odin'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

gondor .ssh # ssh ivan@odin
Last login: Fri Sep 25 15:50:03 PDT 2009 from gondor.artedigital on ssh
Last login: Fri Sep 25 15:50:07 2009 from gondor.artedigital
ivan@odin ~ $

This post is the continuation of  Xvfb and Firefox headless screenshot generator were I explained how Xvfb and Firefox can be used to get a screenshot of any web resource supported by firefox.

The next challenge I had, involved the creation of an automated process that could be called from the web or from a cron job, this process job would take a list of sites that need to be captured, and it will save its shots with their respective thumbnail somewhere, also this same process had to communicate somehow with the front end, to inform the user about the status.

So I’ve planned the architecture for this solution in a modular way

• A daemon checking if new jobs were sent.
• The main process that takes some parameters including the site list and save the generated screenshots somewhere.
• A logger facility to communicate with an external process.
• A web application to take the site list and show the results.

This is the architecture diagram:

From the above modules, I will describe briefly the Main process, this main process started as a simple bash script but this wasn’t sufficient for scalability and security standpoints, so I’ve decided to make it in perl.

 

The current version has changed a lot, but I will describe the main aspects of this script:

The following CPAN extensions were used in this module:

• Run
• DBI
• File::Basename
• Serialize

This process receives the following arguments:

1. The path to a file where the list of urls are, the filename represents the job name and this file must contain 1 url per line (Required)
2. The number of seconds that firefox will have to render each url (Optional)

After the main process is called, either by a daemon or by a cron job, it will connect to a database and search the job based on the received parameter, this job needs to be inserted with an API I did in php, this API is called from the frontend and the cron jobs as well and it serves as a glue point between everything.

After the update is done, I prepare the environment for Xvfb and spawn a new Virtual Screen with its corresponding firefox instance:

This is the code used:

sub initialize {
    logThis("======= Process Starting ==========", 1);
 
    my $numArgs = $#ARGV + 1;
    if ($numArgs < 1) {
        logThis("Bad number of parameters", 1);
        die "Bad Number of params \n";
    }
 
    chomp($siteList = $ARGV[0]);
 
    if ($numArgs > 2) {
        chomp($renderTime = $ARGV[2]);
    }
    ($jobName, my $dirName, my $fileExtension) = fileparse($siteList, ('\.processing') );
    logThis("Parameters received \nSiteList: $siteList\nRelated Domain:$relatedDomain\nRender Time: $renderTime",1);
    logThis("Initializing Db",3);
    $dsn = "DBI:mysql:database=$db{'name'};host=$db{'host'};port=$db{'port'}";
    $dbh = DBI->connect($dsn, $db{'user'}, $db{'pass'}) or die('failed to connect');
    my $jobRow = my $domain = $dbh->selectrow_hashref("SELECT idJobs FROM jobs WHERE jobName = ?",undef,$jobName);
    $jobId = $jobRow->{'idJobs'};
    my $result = $dbh->do("UPDATE jobs SET dateStarted = CURRENT_TIMESTAMP WHERE jobName = ?",undef,$jobName);
 
    logThis("Conected to Db $db{'name'}",1);
    jobUpdate(2, "Starting capture process...");
    $ENV{'DISPLAY'} =':1';
    logThis("Starting virtual screen...",3);
    $pid{'xvfb'} = spawn $xvfbBin, ':1', '-screen', '0', '1024x768x24' or die "Xvfb failed to start $!";
    logThis("Xvfb Pid: $pid{'xvfb'} ",3);
    $pid{'ratpoison'} = spawn 'ratpoison' or die "spawn ratpoison failed";
    logThis("Ratpoison Pid: $pid{'ratpoison'} ",3);
    sleep(1);
    $pid{'firefox'} = spawn $ffBin, '-width 1024', '-height 768' or die "spawn $!";
    logThis("Firefox Process Id: $pid{'firefox'}",1);
    sleep(5);
    logThis("---Initialize sub end here---",4);
    return 1;
}

After the initialization I proceed to read the list of files to start capturing each one, this is the code that does exactly that:

 
while (<FILE>) {
 my $site    = $_;
 chop($site);
 ##
 # Validate that the line starts with http or https
 ##
 if ($site =~ m!\b(https?)://!i) {
    logThis("Processing: $site",4);
    captureSite($site, $relatedDomainId);
    logThis("Sites Processed" . $stats{'processed'} ."=". $stats{'succesfull'} ."+". $stats{'failed'} ."+". $stats{'unrendered'} ."entre ".$stats{'siteCount'},1);
    $stats{'percentage'} = ($stats{'processed'}/$stats{'siteCount'})*100;
    updateStatus('stats', \%stats);
 }
 }

And the captureSite sub is where the magic is, here it is:

sub captureSite {
    my $site          = shift;
    my $result;
    my $imgName;
    my $imageStatus = '';
    my $fileNameExit = 0;
    my $imgCounter = 0;
    my $imgFullName;
    my $thumbFile;
 
    updateStatus('status', "Processing $site");
    jobUpdate(2, "Processing $site");
    logThis("Starting to capture: $site",2);
##
# Check the domain for valid parts
##
    my @domainParts = $site =~ m/\b((?#protocol)https?|ftp):\/\/((?#domain)[-A-Z0-9.]+)((?#file)\/[-A-Z0-9+&@#\/%=~_|!:,.;]*)?((?#parameters)\?[-A-Z0-9+&@#\/%=~_|!:,.;]*)?/ig;
 
    if ($domainParts[0] && $domainParts[1]) {
        if ($openedTabs > $maximumTabs) {
            logThis("Maximum Number of Opened Tabs Reached Killing Firefox pid $pid{'firefox'}",3);
            kill 0, $pid{'firefox'};
            sleep(2);
            $pid{'firefox'} = spawn $ffBin, '-width 1024', '-height 768' or die "spawn $!";
            logThis("New Firefox Process Id: $pid{'firefox'}",1);
            sleep(3);
            $openedTabs = 1;
        }
#because ff is tabless the old page doesn't disappear until
#all content is loaded, so I should remove content first.
        $cmd = $ffBin . ' -remote "openUrl(http://localhost/blank.html)"';
        my $res = `$cmd`;
        sleep(1);
        $cmd = $ffBin . ' -remote "openUrl('.$site.')"';
        logThis("Opening $site",1);
        logThis($cmd, 3);
        $res = `$cmd`;
        $openedTabs++;
        logThis("Waiting $renderTime seconds to allow page rendering",3);
        sleep($renderTime);
##
# We remove harmfull charachters
# maybe an md5sum would be better, because
# we would always have the same number of chars
##
        $imgName = $site;
        $imgName =~ s!\b(https?)://|(www\.)!!g; #removes http://www
            $imgName =~ s!\/|\#|\?$!!g; #removes \ # ? from the end
            $imgName =~ s![^0-9^A-Z^a-z^_^.]!_!g;
#cut excessive large filenames
        if (length($imgName) > $maxFileNameLength) {
            $imgName = substr $imgName, 0, $maxFileNameLength;
        }
        my $baseImgName = $imgName;
        while (!$fileNameExit) {
            $imgFullName = $saveDir . $imgName . $saveFormat;
            $thumbFile   = $thumbsDir . 'thumb_' . $imgName . $saveFormat;
            logThis($imgFullName,1);
            logThis("Cecking if image already exists",1);
            if (-e $imgFullName) {
                $imgCounter++;
                $imgName = $baseImgName . '_' . $imgCounter;
            } else {
                $fileNameExit = 1;
            }
        }
#creating the screenshot
        $cmd = 'import -window root ' . $imgFullName;
        logThis("Command: " .$cmd, 3);
        $res   = `$cmd`;
        if (-e $imgFullName) {
            logThis("The image $imgFullName was created succesfully",1);
            $imageStatus = 'success';
            my $fileSize = -s $imgFullName;
            logThis("File Size for image created: $fileSize",1);
# Check the filesize, if its to low wait give more time to render the page
            if ($fileSize <= $minFileSize) {
                logThis("File is too small, I think the page has not finished rendering, giving it more time...",2);
                my $exit = 0;
                my $try = 0;
                while (!$exit) {
                    logThis("Giving the page $renderTime more seconds to render...",2);
                    updateStatus('status', "Giving $site more time to render...");
                    sleep($renderTime);
                    logThis("Command: " .$cmd, 3);
                    $res   = `$cmd`;
                    $fileSize = -s $imgFullName;
                    logThis("File Size for image created: $fileSize",1);
                    if ($fileSize > $minFileSize) {
                        $exit = 1;
                        $imageStatus = 'success';
                    } else {
                        $try++;
                    }
                    if ($try > 3 ) {
                        $exit = 1;
                        $imageStatus = 'failed';
                    }
                }
            }
            logThis("Croping image...",1);
            $cmd = "convert $imgFullName -crop '1024x768+0+24' $imgFullName";
            $res = `$cmd`;
 
            logThis("Creating Thumbnail in $thumbFile",1);
            $cmd = "convert $imgFullName -resize '170x128' $thumbFile";
            $res = `$cmd`;
            my %image = (
                    site   => $site,
                    image  => $imgFullName,
                    thumb  => $thumbFile,
                    status => $imageStatus,
                    );
            push(@images, \%image);
            insertImage($site, $imgFullName, $thumbFile, $relatedDomainId, $imageStatus);
            if ($imageStatus eq 'success') {
                $stats{'succesfull'}++;
            } else {
                $stats{'failed'}++;
            }
        } else {
            logThis("Fail creating the image",1);
            $stats{'unrendered'}++;
        }
    } else {
        $stats{'unrendered'}++;
    }
    $stats{'processed'}++;
    jobUpdate(2, "Finished processing $site");
}

As you can see I’m updating the job status in the jobUpdate sub, also there are other subs to take care of  inserting the screenshot info in the db.

When this program ends, it updates the db to let the inform the daemon that is ready to take more jobs.

In the next post I will be providing  info on how the frontend works, and what the final results of this implementation ended.

I’ve been working on a project that its main purpose is to get a screenshot of a bunch of different sites. The requirements for this project were simple.

• It has to be fast
• It needs to run on linux

I did some research, trying to found the best approach, I already knew Xvfb but haven’t worked with it in a real project, there wasn’t much info about what I was trying to do, however I’ve found some other automated solutions made in .NET that worked on windows using IE as rendering engine, but I’ve found nothing that worked out of the box in linux

I did found some interesting setups but nothing similar to what I was trying to achieve, and all the directions I’ve got, pointed me to the same place Xvfb, so because I’m a hands-on guy, I quickly cloned a minimal debian box to start playing with this.

First I’ve installed Xvfb, firefox and some fonts

ivano ~ # apt-get install xvfb xfonts-base xfonts-75dpi xfonts-100dpi firefox

After some package downloads I’ve got everything installed, I’ve checked the man page and found:

-screen screennum WxHxD

This option creates screen screennum and sets its width, height,

and depth to W, H, and D respectively. By default, only screen 0

exists and has the dimensions 1280×1024x12.

So I did:

ivano ~ # Xvfb :1 -screen 0 1024x768x24 &
[1] 2665
ivano ~ # Could not init font path element /usr/X11R6/lib/X11/fonts/misc, removing from list!
Could not init font path element /usr/share/fonts/X11/cyrillic, removing from list!
Could not init font path element /usr/share/fonts/X11/Type1, removing from list!
Could not init font path element /usr/X11R6/lib/X11/fonts/Type1, removing from list!
Could not init font path element /var/lib/defoma/x-ttcidfont-conf.d/dirs/TrueType, removing from list!

I’ve got some warnings from X, but nothing lethal, so now I have a Virtual X server ready to throw anything at it.

ivano ~ # DISPLAY=:1 firefox http://google.com &

I have to set the DISPLAY variable to be the same as the virtual screen I’ve created, after this command firefox must be there so I need to get a screen shot of the virtual screen so I’ve used the image magick import tool

ivano ~ # DISPLAY=:1 import -window root google.com.png

When the import program ended I had a the following pg file:

This apparently worked well, however I did some more testing to found any possible problem and the only thing I’ve found was that firefox tried to restore the session because of the improper shutdown so I had to tweak the firefox profile to avoid this, apart from this there were no other significant issues.

I will talk more about how I implemented the backend and frontend for the entire solution.