In this post I will describe the steps needed to setup a central repository for multiple users. I’m using Centos 5.5 as server and debian as a workstation

Create a user account in the server and install git

[root@bp ~]# useradd -m git 
[root@bp ~]# passwd git
[root@bp ~]# yum install git

Copy your public key from our workstation to the server

If you don’t know how to setup an ssh passwordless login, take a look here

ivan@scorpion:~$ scp ~/.ssh/id_rsa.pub git@bp.com:~/ivan.pub

Clone gitolite into the server

ivan@scorpion:~$ ssh git@bp
git@bp: ~$ git clone git://github.com/sitaramc/gitolite
git@bp: ~$ cd gitolite
git@bp: ~$ src/gl-system-install
git@bp: ~$ gl-setup ~/ivan.pub

Go to the workstation to manage the git repo

ivan@scorpion:~$ mkdir -p git/bprepo
ivan@scorpion:~$ cd git/bprepo
ivan@scorpion:~$ git clone git@bp:gitolite-admin

At this point we have 2 folders inside gitolite-admin conf/ and keydir/ the first one is used to create new repositories or add users to repositories the second one is for adding each user public key.

Once you have added new repositories and users you need to do a

git push origin master

on each new repo

The charachters inserted fine, the problem was that I’m using INSERT … ON DUPLICATE KEY UPDATE syntax to avoid looking if the field exists, so after debugging where the application was breaking I got this:

Query failed: Cannot add or update a child row: a foreign key constraint fails 
(`trubaseek/mapKeywordName`, CONSTRAINT `fkDomainMapNameId` FOREIGN KEY (`NameId`) 
REFERENCES `names` (`nameId`) ON DELETE NO ACTION ON UPDATE NO ACTION)<br> 
SQL: INSERT INTO `mapKeywordName` (`keywordId`, `nameId`) VALUES ('1', '326');

so I’ve spent about an hour changing the encodings without success, but after some testing I discovered that hôteles was the same as hotelës, I did several testing from the linux console, the mysql browser, and from a php script, and all revealed the same.

When I did this

INSERT INTO `names` (`name`, `price`, `priceCurrency`, `bids`, `traffic`, `lastUpdate`) 
VALUES ('hotelës', '0', '$US', '0', '0', NOW()) 
ON DUPLICATE KEY UPDATE `price`='0', `priceCurrency`='$US', `bids`='0', `traffic`='0', `lastUpdate` = NOW();

I’ve got this:

Query OK, 2 rows affected (0.01 sec)

Called from a php script mysql_insert_id(); returned me an Id like if it was inserted not updated, but when if I select the returned Id, an error were thrown, because it didn’t existed.

So after some research, I’ve found that adding a collation utf8_bin to that column fixed this issue.

I haven’t tested this enough but for now it appears to be working.

After 2 hours I’ve got a set of regex that worked pretty well, but when I received a new document, the layout was changed a little bit, and my didn’t worked as expected, the content was on the same table but this time it had nested tables, which were a really big problem.

I already have used the PHP DOM, but only on xml files, I didn’t was aware that it had the ability to parse HTML, so I’ve started working with this and within an hour I had it working, and this time the changes in the document didn’t affect the scrapping.

In about an hour I had a larger class with several methods to get all the elements I needed, but suddenly I was presented with another challenge, again with nested tables, sometimes the number of childs were shorter than expected, I’ve experimented several things until I found this Simple Html Dom it is pretty straight forward, and it does an excellent job scrapping html documents, all the methods I did were replaced by this:

            $html   = curl_exec($ch);
            $dom = new simple_html_dom();
            $dom->load($html);
            $items = array();
            $tabla = $dom->find('table[cellpadding^=2]', 0);
 
            foreach ($dom->find('table[cellpadding^=2]') as $table) {
                foreach ($table->find('tr') as $tr) {
                    $link = trim($tr->find('a', 0)->title);
                    if ($link) {
                        $item['item']  = $link;
                        $item['price'] = trim($tr->children(2)->plaintext);
                        $item['bids'] = trim($tr->children(3)->plaintext);
                        $items[] = $item;
                    }
                }
            }