Ivan Villareal » Create a multiuser git repository with gitolite – Ivan Villareal

Create a multiuser git repository with gitolite

Ivan Villareal — Tue, 01 Nov 2011 22:57:54 +0000

I use git for almost all my developments, is fast, it works locally, and I can even use it as a quick deployment tool, by adding hooks, but when it comes to sharing a repository to other developers, I don’t like the idea of creating a user account in the system for each developer, so the easiest and fastest way to solve this issue for me is using gitolite.

In this post I will describe the steps needed to setup a central repository for multiple users. I’m using Centos 5.5 as server and debian as a workstation

Create a user account in the server and install git

[root@bp ~]# useradd -m git 
[root@bp ~]# passwd git
[root@bp ~]# yum install git

Copy your public key from our workstation to the server

If you don’t know how to setup an ssh passwordless login, take a look here

ivan@scorpion:~$ scp ~/.ssh/id_rsa.pub git@bp.com:~/ivan.pub

Clone gitolite into the server

ivan@scorpion:~$ ssh git@bp
git@bp: ~$ git clone git://github.com/sitaramc/gitolite
git@bp: ~$ cd gitolite
git@bp: ~$ src/gl-system-install
git@bp: ~$ gl-setup ~/ivan.pub

Go to the workstation to manage the git repo

ivan@scorpion:~$ mkdir -p git/bprepo
ivan@scorpion:~$ cd git/bprepo
ivan@scorpion:~$ git clone git@bp:gitolite-admin

At this point we have 2 folders inside gitolite-admin conf/ and keydir/ the first one is used to create new repositories or add users to repositories the second one is for adding each user public key.

Once you have added new repositories and users you need to do a

git push origin master

on each new repo

Dynamic multi form on same page with jQuery and PHP

Ivan Villareal — Tue, 19 Apr 2011 20:13:50 +0000

This is a little demo of a small app that allow us to have several forms displayed on the same page, organized by sections, a static multiform can be easily created by putting the form contents on divs and hiding them, then when the user clicks on a specific section we hide the previous div, and show the new one, we can wrap it up everything inside a big form, and submit that.

The approach I did for having this same functionality allows me to:

Save each section form before editing a new one
Change any form inputs depending on received data
Validate in the front and backend each section

The main disadvantage is that for each section I’ll do an ajax call, to submit and retrieve the form.

The first thing I did was to write the main html layout I did a simple 2 column layout to use the left block as a form navigator and the right block to show the actual form.

 class="left_block_box">
	 class="left_block_title">
		 Form Steps
	>
	 id="edit_step_box">
		 width="100%" cellspacing="0" cellpadding="0" border="0" style="font-size: 11px;" id="stepTable">
		 class="stepSelected">
			 class="formStep">
				 Name
			>
			 class="stepEdit">
				 id="formstep1" class="pointer placeEdit">Edit>
			>
		>
		>
			 class="formStep">
				 Address
			>
			 class="stepEdit">
				 id="formstep2" class="pointer placeEdit">Edit>
			>
		>
		>
			 class="formStep">
				 Interests:
			>
			 class="stepEdit">
				 id="formstep3" class="pointer placeEdit">Edit>
			>
		>
		>
	>
>

I’ve used a table so I can navigate more easily the DOM with jquery, but this can be changed an ul and it will work the same, the next thing I wrote was the other column wrapper the right block which will holde the actual form retrieved from the server.

 class="right_block_box">
	 class="right_block_title" id="formTitle">
		 Form 1
	>
	 class="multiform_part">
		 Aqui va la forma
	>
	 id="form_step_nav_wrap">
		 id="form_step_nav">
			 id="form_step_left">
				 type="button" value="Previous Step" id="prevStepBtn"/>
			>
			 id="form_step_right">
				 type="button" value="Next Step" id="nextStepBtn"/>
				 type="button" value="Preview Data" style="display: none" id="previewBtn"/>
			>
		>
	>
>

Once I had my basic layout I started with the javascript functionality, the first thing was to create a function that allow me to move from one form to another, passing current form values to the calling script and displaying the new form.

function selectStep(step) {
	var thisStep = $(step).children().attr('id');
	//get current form data to be passed later
	var curr = $(".stepSelected").children(':eq(1)').children().attr('id');
	var formData = $("#form_"+curr).serialize();
	//add class to selected step 
	$("#stepTable tr").removeClass('stepSelected');
	$(step).parent().addClass('stepSelected');
	//put name of step that is going to be edited 
	$("#formTitle").text($(".stepSelected :eq(0)").text());
	$(".multiform_part").html("Loading...");
	//send data to php, and draw new form
	$.post('processForm.php', {
		step: thisStep, saveData: curr, formData: formData
	}
	,function(data) {
		//draw new form
		$(".multiform_part").html(data);
	}
	);
}

This was easy, then I just binded a click event to each step

	// handle the edit  link on each step
    $(".stepEdit").each(function(index){
        $(this).click(function(e){
            selectStep(this);
            }
        );
    });

And now I have a functional multiform pane, that gets its form from php, the backend was also ver simple to make:

< ?php
session_start();
$step = $_REQUEST['step'];
$formData = $_REQUEST['formData'];
function save($step, $data) {
	//$_SESSION[$step] = $data;
	$fields = explode('&',$data);
	foreach ($fields as $field) {
		$keyVal = explode('=', $field);
		$key = urldecode($keyVal[0]);
		$val = urldecode($keyVal[1]);
		$_SESSION[$step][$key] = $val;
	}
}
if ($formData) {
	$formStep = $_REQUEST['saveData'];
	save($formStep, $_REQUEST['formData']);
}
if ($step) {
	switch ($step) {
		case 'formstep1':
					$html = <<<FORM
					<form id="form_{$step}" name="form_{$step}">
					<fieldset>
		    <legend>Personal informationlegend>
		    <div class="fm-req">
		      <label for="fm-firstname">First name:label>
		      <input name="fm-firstname" id="fm-firstname" type="text" value="{$_SESSION[$step]['fm-firstname']}"/>
		    div>
		    <div class="fm-opt">
		      <label for="fm-middlename">Middle name:label>
		      <input id="fm-middlename" name="fm-middlename" type="text" value="{$_SESSION[$step]['fm-middlename']}"/>
		    div>
		    <div class="fm-req">
		      <label for="fm-lastname">Last name:label>
		      <input name="fm-lastname" id="fm-lastname" type="text" value="{$_SESSION[$step]['fm-lastname']}"/>
		    div>
		    fieldset>
 
 
		FORM;
		echo $html;
		break;
		case 'formstep2':
					$html = < <<FORM
					<form id="form_{$step}" name="form_{$step}">
		<fieldset>
		<legend>Address legend>
		    <div class="fm-opt">
		      <label for="fm-addr">Address:label>
		      <input id="fm-addr" name="fm-addr" type="text" value="{$_SESSION[$step]['fm-addr']}"/>
		    div>
		    <div class="fm-opt">
		      <label for="fm-city">City or Town:label>
		      <input id="fm-city" name="fm-city" type="text" value="{$_SESSION[$step]['fm-city']}"/>
		    div>
		    <div class="fm-opt">
		      <label for="fm-state">State:label>
		      <input id="fm-state" name="fm-state" type="text" value="{$_SESSION[$step]['fm-state']}"/>
		    div>
		    <div class="fm-req">
		      <label for="fm-zipcode">Zip code:label>
		      <input id="fm-zipcode" name="fm-zipcode" type="text" value="{$_SESSION[$step]['fm-zipcode']}"/>
		    div>
		    fieldset>
 
		FORM;
		echo $html;
		break;
		case 'formstep3':
					$html = < <<FORM
					<form id="form_{$step}" name="form_{$step}">
		<fieldset>
		<legend>Interests legend>
		    <div class="fm-opt">
		      <label for="fm-int1">Interest 1:label>
		      <input id="fm-int1" name="fm-int1" type="text" value="{$_SESSION[$step]['fm-int1']}"/>
		    div>
		    <div class="fm-opt">
		      <label for="fm-int2">Interest 2:label>
		      <input id="fm-int2" name="fm-int2" type="text" value="{$_SESSION[$step]['fm-int2']}"/>
		    div>
		    fieldset>
 
		FORM;
		echo $html;
		break;
		case 'getAllData':
					echo 'Saved Data
';
 
		print_r($_SESSION);
 
		break;
	}
}
?>

Finally I’ve added some css styles to make it “pretty” and add some other js functionality to navigate back and forth.

You can checkout the working solution here, or get the files here this demo saves user data in session, and the session is destroyed each time you load the index page.

Script to copy data from MySQL to postgreSQL

Ivan Villareal — Sat, 02 Apr 2011 12:28:32 +0000

This is a perl script I’ve used to move data from mysql 5.1 to postgres 9, this script doesn’t create a schema, it only grabs the data from mysql and try to put it on the same table in postgres.

The only requirement is DBI for mysql and postgres.

 
#!/usr/bin/perl
#
# this script moves all the data from mysql to postgres
# modules required are:
#
# http://search.cpan.org/~timb/DBI-1.616/DBI.pm
#
# please change the dsn, srcUser/srcPass and dstUser/Pass variables.
 
use DBI;
use strict;
 
my $srcUser   = 'ivan';
my $srcPass   = '123';
my $srcDbName = 'datatocopy';
my $srcDbHost = 'mini';
 
my $dstUser  = 'ivan';
my $dstPass  = '456';
my $dstDbName = 'migrateddata';
my $dstDbHost = '192.168.1.145';
 
### do not modify ##
 
my $srcDb  = DBI->connect("dbi:mysql:database=$srcDbName;host=$srcDbHost", $srcUser, $srcPass) or die;
my $dstDb  = DBI->connect("dbi:Pg:database=$dstDbName;host=$dstDbHost", $dstUser, $dstPass) or die;
 
####main########
print "Starting to move data from $srcDbHost to $dstDbHost\n";
$srcDb->do("SET NAMES \'UTF8\'"); 
$dstDb->do("SET CLIENT_ENCODING=\'UTF8\'") or die;
 
 
my $srcTables = $srcDb->selectall_arrayref("SHOW TABLES");
my $dstTables = $dstDb->selectall_hashref("SELECT * FROM pg_tables WHERE NOT tablename ~\'^(pg_|sql_)\'","tablename");
 
my $seqlist = $dstDb->selectall_arrayref("SELECT RELNAME FROM pg_class WHERE relkind=\'S\'");
my $seqmap;
foreach my $s(@$seqlist)  {
    my $seqname = $s->[0];
    if ($seqname=~/^(.*)_([a-z]+)_([a-z]+$)/) {   
        my $item;
        $item->{seq}=$seqname;
        $item->{field}=$2;
        my $table = $1; 
        if (exists $dstTables->{$table}) {   
            push @{$seqmap->{$table}},$item;
        } else {   
            print STDERR "WARN: cannot find table for sequence $seqname\n";
        }   
    }   
}
$dstDb->{AutoCommit}=0;
my $totalRowCount = 0;
foreach my $t(@$srcTables) {  
    my $table = $t->[0];
    unless (exists $dstTables->{$table}) {
        print STDERR "WARN: table $table does not exists in dest db Pg:$dstDbName\n";
        next;
      }
	print "Clearing data from table $table in $dstDbHost\n";
    clear_table($dstDb,$table);
    next if $table=~/^cache/;
    next if $table=~/^locales/;
	print "Copying data from mysql table $table in $srcDbHost to postgres table $table in $dstDbHost\n";
    my $cnt = copy_table($srcDb,$dstDb,$table) unless $table=~/^cache/;
	$totalRowCount = $totalRowCount + $cnt;
    print "$cnt rows copied ($table)\n";
    init_seq($dstDb,$table,$seqmap->{$table}); # if (exists $seqmap->{$table});
  }  
print "Commiting changes \n";     
$dstDb->commit;
print "$totalRowCount rows were copied.\n";
exit 0;
 
 
sub init_seq  {  
    my ($db,$table,$list)=@_;
    return unless $list;
    for my $item (@{$list}) {
        my $ref = $db->selectall_arrayref("SELECT MAX($item->{field}) FROM $table") or die;
        my $val = $ref->[0]->[0];
        if ($val) {
            $db->do("SELECT SETVAL(\'$item->{seq}\',$val)") or die;
          }
      }
}  
 
sub clear_table {  
    my ($dbh,$table)=@_;
    return unless $table;
    $dbh->do("DELETE FROM $table") or die;
}  
 
sub copy_table { 
    my ($srcDb,$dstDb,$table)=@_;
    die unless $table;
    my $slf = $srcDb->prepare("select * from $table limit 1");
    my $rows = $slf->execute() or die;
    return 0 if $rows < 1;
    my $rec1 = $slf->fetchrow_hashref;
    $slf->finish;
    my @fieldnames = sort keys %$rec1;
    my @qm =  map { '?'} @fieldnames;
    my $ins = $dstDb->prepare("INSERT INTO $table (".join(",",@fieldnames).") VALUES(".join(",",@qm).")");
    my $sel = $srcDb->prepare("SELECT * FROM $table");
    $sel->execute or die;
    my $cnt = 0;
    while (my $data = $sel->fetchrow_hashref) {
        $ins->execute(map {$data->{$_}} @fieldnames) or die;
        $cnt++;
      }
    $ins->finish;
    $sel->finish;
    return $cnt;
}

Decrypting SourceCop php files

Ivan Villareal — Wed, 10 Feb 2010 19:41:46 +0000

Every once in a while, I receive code that is encrypted using one of many php encoder software out there, so when I face this situation and if the project is worth it, I start playing around with the encrypted files to see how the encoding algorithm works, so far I’ve been able to successful decrypt files encoded with Zend, and ioncube, I don’t remember the versions of the encoders but I remember that the latter one took me some time, but I did it.

I must say that decoding files also becomes a personal challenge, it is like a hobby when I have the time to play with it.

Yesterday I received a couple of files encoded with an unknown encoder for me, it didn’t require any php modification or extension install, so I tough that it would be easy to break it, because at some point the code must be evaluated, so after I opened the zip file, I noticed a folder called “scopbin“, that contained only 1 php file named “911006.php“, the two encoded files were including this file so I assumed that this is were the decryption logic had to be.

I was exhausted by a long working day, and when I got this files and saw that they were encrypted I placed them in my laptop for later analysis. This analysis was done while I was waiting for the local news, I didn’t research the encoding, or did anything that give me some pointers, I just started to playing around with the code to see how far could I get.

My objective this time was getting this files decrypted, and not analysing the steps of the algorithm, so with this in mind, this is what I did:

This was the original “911006.php” file:

 
< ?php ini_set('include_path',dirname(__FILE__));function A4540acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba,$x276e79316561733d64abdf00f8e8ae48){return $Xew6e79316561733d64abdf00f8e8ae48;}function b5434f0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba,$x276e79316561733d64abdf00f8e8ae48){return $Xew6e79316561733d64abdf00f8e8ae48;}function c43dsd0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba,$x276e79316561733d64abdf00f8e8ae48){return $Xew6e79316561733d64abdf00f8e8ae48;}function Xdsf0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba,$x276e79316561733d64abdf00f8e8ae48){return $Xew6e79316561733d64abdf00f8e8ae48;}function y0666f0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba,$x276e79316561733d64abdf00f8e8ae48){$x0b43c25ccf2340e23492d4d3141479dc='';$x71510c08e23d2083eda280afa650b045=0;$x16754c94f2e48aae0d6f34280507be58=strlen($x897356954c2cd3d41b221e3f24f99bba);$x7a86c157ee9713c34fbd7a1ee40f0c5a=hexdec('&H'.substr($x276e79316561733d64abdf00f8e8ae48,0,2));for($x1b90e1035d4d268e0d8b1377f3dc85a2=2;$x1b90e1035d4d268e0d8b1377f3dc85a2<strlen($x276e79316561733d64abdf00f8e8ae48);$x1b90e1035d4d268e0d8b1377f3dc85a2+=2){$xe594cc261a3b25a9c99ec79da9c91ba5=hexdec(trim(substr($x276e79316561733d64abdf00f8e8ae48, $x1b90e1035d4d268e0d8b1377f3dc85a2, 2)));$x71510c08e23d2083eda280afa650b045=(($x71510c08e23d2083eda280afa650b045<$x16754c94f2e48aae0d6f34280507be58)?$x71510c08e23d2083eda280afa650b045 + 1:1);$xab6389e47b1edcf1a5267d9cfb513ce5=$xe594cc261a3b25a9c99ec79da9c91ba5 ^ ord(substr($x897356954c2cd3d41b221e3f24f99bba, $x71510c08e23d2083eda280afa650b045-1, 1));if($xab6389e47b1edcf1a5267d9cfb513ce5<=$x7a86c157ee9713c34fbd7a1ee40f0c5a)$xab6389e47b1edcf1a5267d9cfb513ce5=255+$xab6389e47b1edcf1a5267d9cfb513ce5-$x7a86c157ee9713c34fbd7a1ee40f0c5a;else $xab6389e47b1edcf1a5267d9cfb513ce5=$xab6389e47b1edcf1a5267d9cfb513ce5-$x7a86c157ee9713c34fbd7a1ee40f0c5a;$x0b43c25ccf2340e23492d4d3141479dc=$x0b43c25ccf2340e23492d4d3141479dc.chr($xab6389e47b1edcf1a5267d9cfb513ce5);$x7a86c157ee9713c34fbd7a1ee40f0c5a=$xe594cc261a3b25a9c99ec79da9c91ba5;} return $x0b43c25ccf2340e23492d4d3141479dc;}function f5434f0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba,$x276e79316561733d64abdf00f8e8ae48){if(file_exists($x456e79316561733d64abdf00f8e8ae48)){unlink($x456e79316561733d64abdf00f8e8ae48);};return $Xew6e79316561733d64abdf00f8e8ae48;}function j43dsd0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba,$x276e79316561733d64abdf00f8e8ae48){if(file_exists($x456e79316561733d64abdf00f8e8ae48)){unlink($x456e79316561733d64abdf00f8e8ae48);};return $Xew6e79316561733d64abdf00f8e8ae48;}function hdsf0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba,$x276e79316561733d64abdf00f8e8ae48){if(file_exists($x456e79316561733d64abdf00f8e8ae48)){unlink($x456e79316561733d64abdf00f8e8ae48);};return $Xew6e79316561733d64abdf00f8e8ae48;}function tr5434f0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba,$x276e79316561733d64abdf00f8e8ae48){if(file_exists($x456e79316561733d64abdf00f8e8ae48)){unlink($x456e79316561733d64abdf00f8e8ae48);};return $Xew6e79316561733d64abdf00f8e8ae48;}function f0666f0acdeed38d4cd9084ade1739498($x) { return implode('',file($x));} function g0666f0acdeed38d4cd9084ade1739498($s){return (strstr($s,'echo')==false?(strstr($s,'print')==false)?(strstr($s,'sprint')==false)?(strstr($s,'sprintf')==false)?false:exit():exit():exit():exit());}function hyr3dsd0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba,$x276e79316561733d64abdf00f8e8ae48){if(file_exists($x456e79316561733d64abdf00f8e8ae48)){unlink($x456e79316561733d64abdf00f8e8ae48);};return $Xew6e79316561733d64abdf00f8e8ae48;}function uygf0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba,$x276e79316561733d64abdf00f8e8ae48){if(file_exists($x456e79316561733d64abdf00f8e8ae48)){unlink($x456e79316561733d64abdf00f8e8ae48);};return $Xew6e79316561733d64abdf00f8e8ae48;}function drfg34f0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba,$x276e79316561733d64abdf00f8e8ae48){if(file_exists($x456e79316561733d64abdf00f8e8ae48)){unlink($x456e79316561733d64abdf00f8e8ae48);};return $Xew6e79316561733d64abdf00f8e8ae48;}function jhkgvdsd0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba,$x276e79316561733d64abdf00f8e8ae48){if(file_exists($x456e79316561733d64abdf00f8e8ae48)){unlink($x456e79316561733d64abdf00f8e8ae48);};return $Xew6e79316561733d64abdf00f8e8ae48;}function yrdhhdacdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba,$x276e79316561733d64abdf00f8e8ae48){if(file_exists($x456e79316561733d64abdf00f8e8ae48)){unlink($x456e79316561733d64abdf00f8e8ae48);};return $Xew6e79316561733d64abdf00f8e8ae48;} ini_set('include_path','.');?>

I’ve used a code formatter to make the code more readable:

ivan@mini:/var/www/copdecrypt/scopbin$ phpCB --space-after-if  \
--space-after-switch                                           \
--space-after-while                                            \
--space-before-srt-angle-bracket                               \
--space-after-end-angle-bracket                                \
--glue-amperscore                                              \
--change-shell-comment-to-double-slashes-comment               \
--force-large-php-code-tag                                     \
--force-true-false-null-contant-lowercase                      \
--align-equal-statements                                       \
--comment-rendering-style PEAR                                 \
--equal-align-position 50                                      \
--padding-char-count 4                                         \
911006.php

And this was the result:

< ?php
ini_set('include_path', dirname(__FILE__));
function A4540acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    return $Xew6e79316561733d64abdf00f8e8ae48;
} 
function b5434f0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    return $Xew6e79316561733d64abdf00f8e8ae48;
} 
function c43dsd0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    return $Xew6e79316561733d64abdf00f8e8ae48;
} 
function Xdsf0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    return $Xew6e79316561733d64abdf00f8e8ae48;
} 
function y0666f0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    $x0b43c25ccf2340e23492d4d3141479dc = '';
    $x71510c08e23d2083eda280afa650b045 = 0;
    $x16754c94f2e48aae0d6f34280507be58 = strlen($x897356954c2cd3d41b221e3f24f99bba);
    $x7a86c157ee9713c34fbd7a1ee40f0c5a = hexdec('&H' . substr($x276e79316561733d64abdf00f8e8ae48, 0, 2));
    for($x1b90e1035d4d268e0d8b1377f3dc85a2 = 2;$x1b90e1035d4d268e0d8b1377f3dc85a2 < strlen($x276e79316561733d64abdf00f8e8ae48);$x1b90e1035d4d268e0d8b1377f3dc85a2 += 2) {
        $xe594cc261a3b25a9c99ec79da9c91ba5 = hexdec(trim(substr($x276e79316561733d64abdf00f8e8ae48, $x1b90e1035d4d268e0d8b1377f3dc85a2, 2)));
        $x71510c08e23d2083eda280afa650b045 = (($x71510c08e23d2083eda280afa650b045 < $x16754c94f2e48aae0d6f34280507be58)?$x71510c08e23d2083eda280afa650b045 + 1:1);
        $xab6389e47b1edcf1a5267d9cfb513ce5 = $xe594cc261a3b25a9c99ec79da9c91ba5 ^ ord(substr($x897356954c2cd3d41b221e3f24f99bba, $x71510c08e23d2083eda280afa650b045-1, 1));
        if ($xab6389e47b1edcf1a5267d9cfb513ce5 <= $x7a86c157ee9713c34fbd7a1ee40f0c5a)$xab6389e47b1edcf1a5267d9cfb513ce5 = 255 + $xab6389e47b1edcf1a5267d9cfb513ce5 - $x7a86c157ee9713c34fbd7a1ee40f0c5a;
        else $xab6389e47b1edcf1a5267d9cfb513ce5 = $xab6389e47b1edcf1a5267d9cfb513ce5 - $x7a86c157ee9713c34fbd7a1ee40f0c5a;
        $x0b43c25ccf2340e23492d4d3141479dc = $x0b43c25ccf2340e23492d4d3141479dc . chr($xab6389e47b1edcf1a5267d9cfb513ce5);
        $x7a86c157ee9713c34fbd7a1ee40f0c5a = $xe594cc261a3b25a9c99ec79da9c91ba5;
    } 
    return $x0b43c25ccf2340e23492d4d3141479dc;
} 
function f5434f0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    if (file_exists($x456e79316561733d64abdf00f8e8ae48)) {
        unlink($x456e79316561733d64abdf00f8e8ae48);
    } ;
    return $Xew6e79316561733d64abdf00f8e8ae48;
} 
function j43dsd0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    if (file_exists($x456e79316561733d64abdf00f8e8ae48)) {
        unlink($x456e79316561733d64abdf00f8e8ae48);
    } ;
    return $Xew6e79316561733d64abdf00f8e8ae48;
} 
function hdsf0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    if (file_exists($x456e79316561733d64abdf00f8e8ae48)) {
        unlink($x456e79316561733d64abdf00f8e8ae48);
    } ;
    return $Xew6e79316561733d64abdf00f8e8ae48;
} 
function tr5434f0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    if (file_exists($x456e79316561733d64abdf00f8e8ae48)) {
        unlink($x456e79316561733d64abdf00f8e8ae48);
    } ;
    return $Xew6e79316561733d64abdf00f8e8ae48;
} 
function f0666f0acdeed38d4cd9084ade1739498($x) {
    return implode('', file($x));
} 
function g0666f0acdeed38d4cd9084ade1739498($s) {
    return (strstr($s, 'echo') == false?(strstr($s, 'print') == false)?(strstr($s, 'sprint') == false)?(strstr($s, 'sprintf') == false)?false:exit():exit():exit():exit());
} 
function hyr3dsd0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    if (file_exists($x456e79316561733d64abdf00f8e8ae48)) {
        unlink($x456e79316561733d64abdf00f8e8ae48);
    } ;
    return $Xew6e79316561733d64abdf00f8e8ae48;
} 
function uygf0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    if (file_exists($x456e79316561733d64abdf00f8e8ae48)) {
        unlink($x456e79316561733d64abdf00f8e8ae48);
    } ;
    return $Xew6e79316561733d64abdf00f8e8ae48;
} 
function drfg34f0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    if (file_exists($x456e79316561733d64abdf00f8e8ae48)) {
        unlink($x456e79316561733d64abdf00f8e8ae48);
    } ;
    return $Xew6e79316561733d64abdf00f8e8ae48;
} 
function jhkgvdsd0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    if (file_exists($x456e79316561733d64abdf00f8e8ae48)) {
        unlink($x456e79316561733d64abdf00f8e8ae48);
    } ;
    return $Xew6e79316561733d64abdf00f8e8ae48;
} 
function yrdhhdacdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    if (file_exists($x456e79316561733d64abdf00f8e8ae48)) {
        unlink($x456e79316561733d64abdf00f8e8ae48);
    } ;
    return $Xew6e79316561733d64abdf00f8e8ae48;
} 
ini_set('include_path', '.');

After a quick review of this file, I saw that it had several functions that had same logic, return an unknow variable or delete the file that this unknown variable had, appart from all this “useless” functions I quickly found what appeared to be the decryption function this was the function named y0666f0acdeed38d4cd9084ade1739498 with this information I headed to check one of the encrypted files, and this is how it looked:

 
< ?php if(!function_exists('findsysfolder')){function findsysfolder($fld){$fld1=dirname($fld);$fld=$fld1.'/scopbin';clearstatcache();if(!is_dir($fld))return findsysfolder($fld1);else return $fld;}}require_once(findsysfolder(__FILE__).'/911006.php');$REXISTHECAT4FBI='FE50E574D754E76AC679F242F450F768FB5DCB77F34DE341 660C280D176E374DE7FB3B090A782B6B68DBC97BEAD93B681C452F25BE26';g0666f0acdeed38d4cd9084ade1739498(f0666f0acdeed38d4cd9084ade1739498(__FILE__));$REXISTHEDOG4FBI='9CEF6BE117B329ADFC4560538EBF16BB6DAD1748FE354E8EDA7AABFA376EB6938496F43560E4123D85D272E342E363FD51F3181763A3F623 660B6E2369243EE5781CD76A133E272E37DACA198968397F150425B4326A126CD64F051F35287DE7BD3 5354276B880BDEF525247533610 64282D07BA63861F81DB715C115BC1BD57FC5D9C8 225D2 A F177EDC7EA026A7E74D37AC28A125D1B8F73B63C6D1C3A1E137639BEECEDACFDBBD9F8D948CE93D6FCDCDD8CCA98B36AA21A08B8BD87FD8 15143C2C2D37593B82857516FC9ABF9237BDEC9DBB9F92F7BDEB8EE3B6AEA153C9DF3 A2CAE2CD218B1E566C0D274D6 F32DC2A E20 512 062A2F72C 96FB9ED3B6FC9ABF927A34985CAB5D3 65D89B315B3E61076D8 E4488B2127FCEF82C78DDBBED3A6FBB7FCEEACEE04453412363B6E246F05086C7 52E C408CC263E078A53461E0 331B71EB81DC475DB6CAC9531914393D4 22AC9C3D7CE7194438AD638 03D6FCDCDD8CCA98F99211B1177D5C7DEC6A3F427 5 510 461C7D1C51AB492F850FA56FE445EC0154885BDACDC 819BD76A72A419A4D859CFE5A5268CDABFD2A7FAB8DEF227BA3869183E12277A386E03763B6E241F75282CDBFCA187ECE74D7 D1D4881D3C7432177D4 E3F83DA54417EAA8FE93C68BDE94C2E7CA5FD5B4C5E3C9993A98CEA3D69BCE84F2D989C9EEECFDDBFFF15 164C2D471D3 228D3 51CBA75D4 A3C85C1 4618EC5 E6BE91FCD F4D87BE F 56BFB5BFE2A3EA923BDDDBB9D26BE13CD 41F6E584D593C1A C18 D19D29D958C94F12577D5D5C0D4B19781958094F21EB6EF20CC77D87CD0CDC57FA13D8484F61076D47ED5 366F521A6FA2FEFBAF22772A680E23168B0958290F23366B297F12470A5F1543664EF6DA42DAD5E5EB718C569EA2C5F8AE81BBF16BE489E3EE13D7FBD98FE297DA8FC5B396BB2EA4E594B2969D3C7689230A1F4441763B1E83114 31173B3E6331670A6F22672D4B6E43C64C1D6C475FA5AFB2BDF36A83F9F55F8 62A C69BEEC4F4F5A4E2B D1B F1A E68CAD86F44 B1C E6CACF92E B6DDCDDD26437 3';$REXISTHECAT4FBI='94CD76CD371C5A7BC70C186E779C293B9B49BACA5A781A6'; eval(y0666f0acdeed38d4cd9084ade1739498('4EF6454FB298E72B 5',$REXISTHEDOG4FBI));?>

Again I used the code beautifier to make this file more readable:

< ?php if (!function_exists('findsysfolder')) {
    function findsysfolder($fld) {
        $fld1 = dirname($fld);
        $fld = $fld1 . '/scopbin';
        clearstatcache();
        if (!is_dir($fld))return findsysfolder($fld1);
        else return $fld;
    } 
} 
require_once(findsysfolder(__FILE__) . '/911006.php');
$REXISTHECAT4FBI = 'FE50E574D754E76AC679F242F450F768FB5DCB77F34DE341 660C280D176E374DE7FB3B090A782B6B68DBC97BEAD93B681C452F25BE26';
g0666f0acdeed38d4cd9084ade1739498(f0666f0acdeed38d4cd9084ade1739498(__FILE__));
$REXISTHEDOG4FBI = '9CEF6BE117B329ADFC4560538EBF16BB6DAD1748FE354E8EDA7AABFA376EB6938496F43560E4123D85D272E342E363FD51F3181763A3F623 660B6E2369243EE5781CD76A133E272E37DACA198968397F150425B4326A126CD64F051F35287DE7BD3 5354276B880BDEF525247533610 64282D07BA63861F81DB715C115BC1BD57FC5D9C8 225D2 A F177EDC7EA026A7E74D37AC28A125D1B8F73B63C6D1C3A1E137639BEECEDACFDBBD9F8D948CE93D6FCDCDD8CCA98B36AA21A08B8BD87FD8 15143C2C2D37593B82857516FC9ABF9237BDEC9DBB9F92F7BDEB8EE3B6AEA153C9DF3 A2CAE2CD218B1E566C0D274D6 F32DC2A E20 512 062A2F72C 96FB9ED3B6FC9ABF927A34985CAB5D3 65D89B315B3E61076D8 E4488B2127FCEF82C78DDBBED3A6FBB7FCEEACEE04453412363B6E246F05086C7 52E C408CC263E078A53461E0 331B71EB81DC475DB6CAC9531914393D4 22AC9C3D7CE7194438AD638 03D6FCDCDD8CCA98F99211B1177D5C7DEC6A3F427 5 510 461C7D1C51AB492F850FA56FE445EC0154885BDACDC 819BD76A72A419A4D859CFE5A5268CDABFD2A7FAB8DEF227BA3869183E12277A386E03763B6E241F75282CDBFCA187ECE74D7 D1D4881D3C7432177D4 E3F83DA54417EAA8FE93C68BDE94C2E7CA5FD5B4C5E3C9993A98CEA3D69BCE84F2D989C9EEECFDDBFFF15 164C2D471D3 228D3 51CBA75D4 A3C85C1 4618EC5 E6BE91FCD F4D87BE F 56BFB5BFE2A3EA923BDDDBB9D26BE13CD 41F6E584D593C1A C18 D19D29D958C94F12577D5D5C0D4B19781958094F21EB6EF20CC77D87CD0CDC57FA13D8484F61076D47ED5 366F521A6FA2FEFBAF22772A680E23168B0958290F23366B297F12470A5F1543664EF6DA42DAD5E5EB718C569EA2C5F8AE81BBF16BE489E3EE13D7FBD98FE297DA8FC5B396BB2EA4E594B2969D3C7689230A1F4441763B1E83114 31173B3E6331670A6F22672D4B6E43C64C1D6C475FA5AFB2BDF36A83F9F55F8 62A C69BEEC4F4F5A4E2B D1B F1A E68CAD86F44 B1C E6CACF92E B6DDCDDD26437 3';
$REXISTHECAT4FBI = '94CD76CD371C5A7BC70C186E779C293B9B49BACA5A781A6';
eval(y0666f0acdeed38d4cd9084ade1739498('4EF6454FB298E72B 5', $REXISTHEDOG4FBI));

So what we have here is more obfuscated code, but a simple to understand, we have 2 variables and 3 functions, the variable that has the encrypted code should be the larger one so I'm assuming that is the "$REXISTHEDOG4FBI" variable, the other one apparently isn't used, so what I did at this point was to print the results of the evaluated function which corresponds to the function where the decrypt logic is (deofuscated):

function ($key, $program) {
    $result = '';
    $position = 0;
    $keyLength = strlen($key);
    $decValue = hexdec('&H' . substr($program, 0, 2));
 
    for($i = 2;$i < strlen($program);$i += 2) {
        $decProgram = hexdec(trim(substr($program, $i, 2)));
        $position   = (($position < $keyLength) ? $position + 1 : 1);
        $ascii = $decProgram^ ord(substr($key, $position-1, 1));
 
        if ($ascii <= $decValue) $ascii = 255 + $ascii - $decValue;
        else $ascii = $ascii - $decValue;
 
        $result   = $result . chr($ascii);
        $decValue = $decProgram;
    } 
    return $result;
}

unfortunately printing the result didn't work, so I did a review to check why, and I found that the other 2 functions where the problem, let me explain what happens, first a call is made to

function f0666f0acdeed38d4cd9084ade1739498($x) {
 
    return implode('', file($x));
 
}

this function receives the name of the executing file (test.php in my case), read its content in an array, then glues all lines to create a 1 line string, without any new lines, then this result is passed to the following function:

function g0666f0acdeed38d4cd9084ade1739498($s) {
 
    return (strstr($s, 'echo') == false ? (strstr($s, 'print') == false) ? (strstr($s, 'sprint') == false) ? (strstr($s, 'sprintf') == false) ? false : exit() : exit() : exit() : exit());
 
}

which are a series of nested ternary conditions, looking for the words "echo,print,sprint,sprintf", if any of this words are in the encrypted file, then the script simply exits, that's why I was unable to print the decrypted code after calling the decoding function, so simply avoiding the call to this function will fix the issue and I will get my decrypted code.
However, I didn't want to modify the original encrypted files, so a second approach I took was modifying the decryption function and echoing the output from there, so I've added a couple of lines (15 and 16) to the "y0666f0acdeed38d4cd9084ade1739498" function:

function y0666f0acdeed38d4cd9084ade1739498($x897356954c2cd3d41b221e3f24f99bba, $x276e79316561733d64abdf00f8e8ae48) {
    $x0b43c25ccf2340e23492d4d3141479dc = '';
    $x71510c08e23d2083eda280afa650b045 = 0;
    $x16754c94f2e48aae0d6f34280507be58 = strlen($x897356954c2cd3d41b221e3f24f99bba);
    $x7a86c157ee9713c34fbd7a1ee40f0c5a = hexdec('&H' . substr($x276e79316561733d64abdf00f8e8ae48, 0, 2));
    for($x1b90e1035d4d268e0d8b1377f3dc85a2 = 2;$x1b90e1035d4d268e0d8b1377f3dc85a2 < strlen($x276e79316561733d64abdf00f8e8ae48);$x1b90e1035d4d268e0d8b1377f3dc85a2 += 2) {
        $xe594cc261a3b25a9c99ec79da9c91ba5 = hexdec(trim(substr($x276e79316561733d64abdf00f8e8ae48, $x1b90e1035d4d268e0d8b1377f3dc85a2, 2)));
        $x71510c08e23d2083eda280afa650b045 = (($x71510c08e23d2083eda280afa650b045 < $x16754c94f2e48aae0d6f34280507be58)?$x71510c08e23d2083eda280afa650b045 + 1:1);
        $xab6389e47b1edcf1a5267d9cfb513ce5 = $xe594cc261a3b25a9c99ec79da9c91ba5 ^ ord(substr($x897356954c2cd3d41b221e3f24f99bba, $x71510c08e23d2083eda280afa650b045-1, 1));
        if ($xab6389e47b1edcf1a5267d9cfb513ce5 <= $x7a86c157ee9713c34fbd7a1ee40f0c5a)$xab6389e47b1edcf1a5267d9cfb513ce5 = 255 + $xab6389e47b1edcf1a5267d9cfb513ce5 - $x7a86c157ee9713c34fbd7a1ee40f0c5a;
        else $xab6389e47b1edcf1a5267d9cfb513ce5 = $xab6389e47b1edcf1a5267d9cfb513ce5 - $x7a86c157ee9713c34fbd7a1ee40f0c5a;
        $x0b43c25ccf2340e23492d4d3141479dc = $x0b43c25ccf2340e23492d4d3141479dc . chr($xab6389e47b1edcf1a5267d9cfb513ce5);
        $x7a86c157ee9713c34fbd7a1ee40f0c5a = $xe594cc261a3b25a9c99ec79da9c91ba5;
    } 
    echo $x0b43c25ccf2340e23492d4d3141479dc;
    die();
    return $x0b43c25ccf2340e23492d4d3141479dc;
}

And voilà, I was able to see the source code.

So in short, just print the output of the decrypt function, and kill the script to get the decrypted code, I don't know if there are any other versions of the "911006.php" file, but I guess that the same logic applies.

File Type Detection in PHP

Ivan Villareal — Fri, 25 Dec 2009 04:47:24 +0000

I’m working on a project where the users can upload files, and I need to know the type of file they are uploading, a simple solution would be to check the extensions of the files, but this has many flaws, so I was looking for a reliable way to detect the type of file, and I found a PECL extension called file info this extension is enabled by default in PHP 5.3, unfortunately I’m runing PHP 5.2.10, and the prod server also has this version.

So what I did to install this on ubuntu 9.10 karmic was the following:

root@mini:/etc/php5/apache2# apt-get install php5-dev libmagic-dev php-pear
....
root@mini:/etc/php5/apache2# pear channel-update pear.php.net
...
root@mini:/etc/php5/apache2# pecl install Fileinfo
....
Build process completed successfully
Installing '/usr/lib/php5/20060613+lfs/fileinfo.so'
install ok: channel://pear.php.net/Fileinfo-1.0.4

and finally I added the extension to php.ini

extension=fileinfo.so

I restarted the apache server, and tested with this

$fileInfo = new finfo(FILEINFO_MIME); 
$mimeType = $fileInfo->buffer(file_get_contents($uploadedFile));

Format a Mysql DateTime Field with Zend_Date

Ivan Villareal — Thu, 10 Dec 2009 19:54:41 +0000

Zend_Date is really a time saver class, I use it in almost every project I work, It handles time_zones, and date formatting pretty easy, leaving me more time to focus on other things.

The files that I need to use Zend_Date as a standalone class are:

Date.php and the Date folder from the Zend distribution package, and this is how I use it:

&lt; ?php
set_include_path('./lib' . PATH_SEPARATOR . get_include_path());
include_once 'Zend/Date.php';
$tugId = isset($_GET['tugId']) ? (int) $_GET['tugId'] : false;
 
function formatDate($date) {
    /**
     * List of supported timezones
     * http://php.net/manual/en/timezones.php
     */
    if ($date) {
        $fmtDate = new Zend_Date($date, Zend_Date::ISO_8601);
        $fmtDate-&gt;setTimezone('America/Los_Angeles');
        return $fmtDate-&gt;toString(Zend_Date::DATETIME);
    } else {
        return '';
    }
}
 
if (is_int($tugId)) {
 $sql = 'SELECT modified_date FROM tuggler WHERE tug_id = ' . $tugId;
 $result = $db-&gt;fetchAssoc($sql);
 $response = 'Last Record Update: '.formatDate($row['modified_date']) .'';
} else {
 $response "nothing to return";
}
 
echo $response;

The response from the above would be Jul 9, 2009 11:41:20 AM but I can easily change it using one of the predefined constants, or I can make my own format.

Filling rtf templates with text and images using php

Ivan Villareal — Wed, 11 Nov 2009 18:10:22 +0000

We had a requirement to fill an rtf template file with some info from the system, the templates had some pictures that need to be retrieved from the system and placed in the rtf file.

The deadline for this requirement was very short, and the system where this was implemented was a large crm system so we didn’t had much time to dig up into the code to look for any useful class, so what we did was very simple.

Whenever a new document would be created, we get the template related to that client opened the rtf file put it on a string and do a string replace for each field to be replaced, this is the code for this:

$this->keyVaribles = array
 (
 '' => $account->name,
 ''     => $img->getContent(),
 ''    => $account->date,
 ''                     => $account->link,
 ''              => $account->cal,
 ''             => $account->signature,
 );
 
// read file
$file = $bean->document_revision_id;
$handle = fopen ($file, 'r');
$content = fread($handle, filesize($file));
 
// replace content
foreach($this->keyVaribles as $from=> $to) {
   $content = str_replace($from, $to, $content);
}
 
// save file
file_put_contents($file, $content);
fclose($handle);signature>calendar>link>dates_of_event>company_logos>organization_name>

This worked for all the text fields, but there was one field that needed an image to be inserted, I did little research on this, and used and old library (http://www.phprtf.com/) this didn’t worked very well, because it had only methods to save or send the generated rtf file, I needed only the image to be inserted so I did a class to convert an image into an rtf embeddable code, here it is:

< ?php
 
/**
 *
 * This is the short Description for the Class
 *
 * This is the long description for the Class
 *
 * @author		Ivan Villareal
 * @version		$Id$
 * @package		Package
 * @subpackage	SubPackage
 *
 */
class image
{
	private $_cmInPoints = 0.026434;
	private $_twipsInCm  = 567;
	private $_imgFile;
	private $_width;
	private $_height;
	private $_defaultWidth;
	private $_defaultHeight;
 
	public function __construct($fileName, $width = 0, $height = 0)
	{
		$this->_imgFile = $fileName;
		$this->_width   = $width;
		$this->_height	= $height;
		$imgSize = getimagesize($fileName);
		$this->_defaultWidth  = $imgSize[0];
		$this->_defaultHeight = $imgSize[1];
	}
 
	private function _getWidth()
	{	
		if (!empty($this->_width)) {
			$width = $this->_width;
		} else if (!empty($this->_height)) {
			$width = ($this->_defaultWidth / $this->_defaultHeight) * $this->_height;
		} else {
			$width = $this->_defaultWidth * $this->_cmInPoints;
		}
		return round($width * $this->_twipsInCm);
	}
 
	private function _getHeight()
	{	
		if (!empty($this->_height)) {
			$height = $this->_height;
		} else if (!empty($this->_height)) {
			$height = ($this->_defaultHeight / $this->_defaultWidth) * $this->_width; 
		} else {
			$height = $this->_defaultHeight * $this->_cmInPoints;
		}
		return round($height * $this->_twipsInCm);
	}
 
 
	private	function _fileToHex() 
	{	  
	  	$f = fopen($this->_imgFile, "r");
		$string = fread($f, filesize($this->_imgFile));
		fclose($f);
 
		$str = '';
		for ($i = 0; $i < strlen($string); $i ++) {
			$hex = dechex( ord($string{$i}));
 
			if (strlen($hex) == 1) {			  
			  	$hex = '0'.$hex;
			}
 
			$str .= $hex;	
		}
 
		return $str;
	}
 
	public function getContent() 
	{	  
		$content = '{\pict';
		$content .= '\picwgoal'.$this->_getWidth();  		
		$content .= '\pichgoal'.$this->_getHeight();  
		$content .= $this->_fileToHex();		
		$content .= '}'; 			
		return $content;
	}
}

As you can see this class is very simple and you only have to instantiate it with the image file to be converted, and call the getContents method to get the rtf code for that image.

This worked pretty well so far, but I guess we need to do some more testing before deploying this.

Mysql unique keys and collations

Ivan Villareal — Thu, 08 Oct 2009 01:10:59 +0000

I’m working on an application that stores unique values on a field, so I used the unique index for this column, everything was ok, until I’ve started inserting multi byte characters in it.

The charachters inserted fine, the problem was that I’m using INSERT … ON DUPLICATE KEY UPDATE syntax to avoid looking if the field exists, so after debugging where the application was breaking I got this:

Query failed: Cannot add or update a child row: a foreign key constraint fails 
(`trubaseek/mapKeywordName`, CONSTRAINT `fkDomainMapNameId` FOREIGN KEY (`NameId`) 
REFERENCES `names` (`nameId`) ON DELETE NO ACTION ON UPDATE NO ACTION)<br> 
SQL: INSERT INTO `mapKeywordName` (`keywordId`, `nameId`) VALUES ('1', '326');

so I’ve spent about an hour changing the encodings without success, but after some testing I discovered that hôteles was the same as hotelës, I did several testing from the linux console, the mysql browser, and from a php script, and all revealed the same.

When I did this

INSERT INTO `names` (`name`, `price`, `priceCurrency`, `bids`, `traffic`, `lastUpdate`) 
VALUES ('hotelës', '0', '$US', '0', '0', NOW()) 
ON DUPLICATE KEY UPDATE `price`='0', `priceCurrency`='$US', `bids`='0', `traffic`='0', `lastUpdate` = NOW();

I’ve got this:

Query OK, 2 rows affected (0.01 sec)

Called from a php script mysql_insert_id(); returned me an Id like if it was inserted not updated, but when if I select the returned Id, an error were thrown, because it didn’t existed.

So after some research, I’ve found that adding a collation utf8_bin to that column fixed this issue.

I haven’t tested this enough but for now it appears to be working.

Parsing HTML documents with Simple Html DOM

Ivan Villareal — Wed, 30 Sep 2009 23:02:34 +0000

I had to get some information from an XHTML document, the information required was on nested tables, I’ve started working on that and my first approach was to get the needed info using regular expressions.

After 2 hours I’ve got a set of regex that worked pretty well, but when I received a new document, the layout was changed a little bit, and my didn’t worked as expected, the content was on the same table but this time it had nested tables, which were a really big problem.

I already have used the PHP DOM, but only on xml files, I didn’t was aware that it had the ability to parse HTML, so I’ve started working with this and within an hour I had it working, and this time the changes in the document didn’t affect the scrapping.

In about an hour I had a larger class with several methods to get all the elements I needed, but suddenly I was presented with another challenge, again with nested tables, sometimes the number of childs were shorter than expected, I’ve experimented several things until I found this Simple Html Dom it is pretty straight forward, and it does an excellent job scrapping html documents, all the methods I did were replaced by this:

            $html   = curl_exec($ch);
            $dom = new simple_html_dom();
            $dom->load($html);
            $items = array();
            $tabla = $dom->find('table[cellpadding^=2]', 0);
 
            foreach ($dom->find('table[cellpadding^=2]') as $table) {
                foreach ($table->find('tr') as $tr) {
                    $link = trim($tr->find('a', 0)->title);
                    if ($link) {
                        $item['item']  = $link;
                        $item['price'] = trim($tr->children(2)->plaintext);
                        $item['bids'] = trim($tr->children(3)->plaintext);
                        $items[] = $item;
                    }
                }
            }

Automated Capture of Webpages (Main Process)

Ivan Villareal — Fri, 25 Sep 2009 00:21:45 +0000

This post is the continuation of Xvfb and Firefox headless screenshot generator were I explained how Xvfb and Firefox can be used to get a screenshot of any web resource supported by firefox.

The next challenge I had, involved the creation of an automated process that could be called from the web or from a cron job, this process job would take a list of sites that need to be captured, and it will save its shots with their respective thumbnail somewhere, also this same process had to communicate somehow with the front end, to inform the user about the status.

So I’ve planned the architecture for this solution in a modular way

A daemon checking if new jobs were sent.
The main process that takes some parameters including the site list and save the generated screenshots somewhere.
A logger facility to communicate with an external process.
A web application to take the site list and show the results.

This is the architecture diagram:

Screen Grabber System Arquitechture

From the above modules, I will describe briefly the Main process, this main process started as a simple bash script but this wasn’t sufficient for scalability and security standpoints, so I’ve decided to make it in perl.

The current version has changed a lot, but I will describe the main aspects of this script:

The following CPAN extensions were used in this module:

This process receives the following arguments:

The path to a file where the list of urls are, the filename represents the job name and this file must contain 1 url per line (Required)
The number of seconds that firefox will have to render each url (Optional)

After the main process is called, either by a daemon or by a cron job, it will connect to a database and search the job based on the received parameter, this job needs to be inserted with an API I did in php, this API is called from the frontend and the cron jobs as well and it serves as a glue point between everything.

After the update is done, I prepare the environment for Xvfb and spawn a new Virtual Screen with its corresponding firefox instance:

This is the code used:

sub initialize {
    logThis("======= Process Starting ==========", 1);
 
    my $numArgs = $#ARGV + 1;
    if ($numArgs < 1) {
        logThis("Bad number of parameters", 1);
        die "Bad Number of params \n";
    }
 
    chomp($siteList = $ARGV[0]);
 
    if ($numArgs > 2) {
        chomp($renderTime = $ARGV[2]);
    }
    ($jobName, my $dirName, my $fileExtension) = fileparse($siteList, ('\.processing') );
    logThis("Parameters received \nSiteList: $siteList\nRelated Domain:$relatedDomain\nRender Time: $renderTime",1);
    logThis("Initializing Db",3);
    $dsn = "DBI:mysql:database=$db{'name'};host=$db{'host'};port=$db{'port'}";
    $dbh = DBI->connect($dsn, $db{'user'}, $db{'pass'}) or die('failed to connect');
    my $jobRow = my $domain = $dbh->selectrow_hashref("SELECT idJobs FROM jobs WHERE jobName = ?",undef,$jobName);
    $jobId = $jobRow->{'idJobs'};
    my $result = $dbh->do("UPDATE jobs SET dateStarted = CURRENT_TIMESTAMP WHERE jobName = ?",undef,$jobName);
 
    logThis("Conected to Db $db{'name'}",1);
    jobUpdate(2, "Starting capture process...");
    $ENV{'DISPLAY'} =':1';
    logThis("Starting virtual screen...",3);
    $pid{'xvfb'} = spawn $xvfbBin, ':1', '-screen', '0', '1024x768x24' or die "Xvfb failed to start $!";
    logThis("Xvfb Pid: $pid{'xvfb'} ",3);
    $pid{'ratpoison'} = spawn 'ratpoison' or die "spawn ratpoison failed";
    logThis("Ratpoison Pid: $pid{'ratpoison'} ",3);
    sleep(1);
    $pid{'firefox'} = spawn $ffBin, '-width 1024', '-height 768' or die "spawn $!";
    logThis("Firefox Process Id: $pid{'firefox'}",1);
    sleep(5);
    logThis("---Initialize sub end here---",4);
    return 1;
}

After the initialization I proceed to read the list of files to start capturing each one, this is the code that does exactly that:

while () {
 my $site    = $_;
 chop($site);
 ##
 # Validate that the line starts with http or https
 ##
 if ($site =~ m!\b(https?)://!i) {
    logThis("Processing: $site",4);
    captureSite($site, $relatedDomainId);
    logThis("Sites Processed" . $stats{'processed'} ."=". $stats{'succesfull'} ."+". $stats{'failed'} ."+". $stats{'unrendered'} ."entre ".$stats{'siteCount'},1);
    $stats{'percentage'} = ($stats{'processed'}/$stats{'siteCount'})*100;
    updateStatus('stats', \%stats);
 }
 }file>

And the captureSite sub is where the magic is, here it is:

sub captureSite {
    my $site          = shift;
    my $result;
    my $imgName;
    my $imageStatus = '';
    my $fileNameExit = 0;
    my $imgCounter = 0;
    my $imgFullName;
    my $thumbFile;
 
    updateStatus('status', "Processing $site");
    jobUpdate(2, "Processing $site");
    logThis("Starting to capture: $site",2);
##
# Check the domain for valid parts
##
    my @domainParts = $site =~ m/\b((?#protocol)https?|ftp):\/\/((?#domain)[-A-Z0-9.]+)((?#file)\/[-A-Z0-9+&@#\/%=~_|!:,.;]*)?((?#parameters)\?[-A-Z0-9+&@#\/%=~_|!:,.;]*)?/ig;
 
    if ($domainParts[0] && $domainParts[1]) {
        if ($openedTabs > $maximumTabs) {
            logThis("Maximum Number of Opened Tabs Reached Killing Firefox pid $pid{'firefox'}",3);
            kill 0, $pid{'firefox'};
            sleep(2);
            $pid{'firefox'} = spawn $ffBin, '-width 1024', '-height 768' or die "spawn $!";
            logThis("New Firefox Process Id: $pid{'firefox'}",1);
            sleep(3);
            $openedTabs = 1;
        }
#because ff is tabless the old page doesn't disappear until
#all content is loaded, so I should remove content first.
        $cmd = $ffBin . ' -remote "openUrl(http://localhost/blank.html)"';
        my $res = `$cmd`;
        sleep(1);
        $cmd = $ffBin . ' -remote "openUrl('.$site.')"';
        logThis("Opening $site",1);
        logThis($cmd, 3);
        $res = `$cmd`;
        $openedTabs++;
        logThis("Waiting $renderTime seconds to allow page rendering",3);
        sleep($renderTime);
##
# We remove harmfull charachters
# maybe an md5sum would be better, because
# we would always have the same number of chars
##
        $imgName = $site;
        $imgName =~ s!\b(https?)://|(www\.)!!g; #removes http://www
            $imgName =~ s!\/|\#|\?$!!g; #removes \ # ? from the end
            $imgName =~ s![^0-9^A-Z^a-z^_^.]!_!g;
#cut excessive large filenames
        if (length($imgName) > $maxFileNameLength) {
            $imgName = substr $imgName, 0, $maxFileNameLength;
        }
        my $baseImgName = $imgName;
        while (!$fileNameExit) {
            $imgFullName = $saveDir . $imgName . $saveFormat;
            $thumbFile   = $thumbsDir . 'thumb_' . $imgName . $saveFormat;
            logThis($imgFullName,1);
            logThis("Cecking if image already exists",1);
            if (-e $imgFullName) {
                $imgCounter++;
                $imgName = $baseImgName . '_' . $imgCounter;
            } else {
                $fileNameExit = 1;
            }
        }
#creating the screenshot
        $cmd = 'import -window root ' . $imgFullName;
        logThis("Command: " .$cmd, 3);
        $res   = `$cmd`;
        if (-e $imgFullName) {
            logThis("The image $imgFullName was created succesfully",1);
            $imageStatus = 'success';
            my $fileSize = -s $imgFullName;
            logThis("File Size for image created: $fileSize",1);
# Check the filesize, if its to low wait give more time to render the page
            if ($fileSize < = $minFileSize) {
                logThis("File is too small, I think the page has not finished rendering, giving it more time...",2);
                my $exit = 0;
                my $try = 0;
                while (!$exit) {
                    logThis("Giving the page $renderTime more seconds to render...",2);
                    updateStatus('status', "Giving $site more time to render...");
                    sleep($renderTime);
                    logThis("Command: " .$cmd, 3);
                    $res   = `$cmd`;
                    $fileSize = -s $imgFullName;
                    logThis("File Size for image created: $fileSize",1);
                    if ($fileSize > $minFileSize) {
                        $exit = 1;
                        $imageStatus = 'success';
                    } else {
                        $try++;
                    }
                    if ($try > 3 ) {
                        $exit = 1;
                        $imageStatus = 'failed';
                    }
                }
            }
            logThis("Croping image...",1);
            $cmd = "convert $imgFullName -crop '1024x768+0+24' $imgFullName";
            $res = `$cmd`;
 
            logThis("Creating Thumbnail in $thumbFile",1);
            $cmd = "convert $imgFullName -resize '170x128' $thumbFile";
            $res = `$cmd`;
            my %image = (
                    site   => $site,
                    image  => $imgFullName,
                    thumb  => $thumbFile,
                    status => $imageStatus,
                    );
            push(@images, \%image);
            insertImage($site, $imgFullName, $thumbFile, $relatedDomainId, $imageStatus);
            if ($imageStatus eq 'success') {
                $stats{'succesfull'}++;
            } else {
                $stats{'failed'}++;
            }
        } else {
            logThis("Fail creating the image",1);
            $stats{'unrendered'}++;
        }
    } else {
        $stats{'unrendered'}++;
    }
    $stats{'processed'}++;
    jobUpdate(2, "Finished processing $site");
}

As you can see I’m updating the job status in the jobUpdate sub, also there are other subs to take care of inserting the screenshot info in the db.

When this program ends, it updates the db to let the inform the daemon that is ready to take more jobs.

In the next post I will be providing info on how the frontend works, and what the final results of this implementation ended.