Ivan Villareal

get Name and email address from a “from” email header

Ivan Villareal — Wed, 16 Jun 2010 18:21:30 +0000

This is a quick post of a little function I did to get the name and the email of a “from” header, Because this is not a standard thing this function only recognizes the following format:

Some Name

this is the function I came up:

function extractEmailAddress($subject)
    {
        $result = new stdClass();
        if (preg_match('/\\b[A-Z0-9._%-]+@[A-Z0-9.-]+\\.[A-Z]{2,4}\\b/i', $subject, $regs)) {
            $result->email = $regs[0];
        }
        //try to get the name
        preg_match_all('/(.*)< (.*)>/', $subject, $regs, PREG_PATTERN_ORDER);
        if (isset($regs[1][0])) {
            $result->name = ($regs[1][0] != '') ? $regs[1][0] : '';
        } else {
            $result->name = '';
        }
        return $result;
    }

And this are some results I had:

Case 1:

$testName = "Ivan Villareal ";  ivan>

Result:

object(stdClass)#3 (2) {
  ["email"]=>
  string(15) "ivan@server.com"
  ["name"]=>
  string(15) "Ivan Villareal "
}

Case 2:

$testName = "Ivan Villareal ivan@server.com";

Result:

object(stdClass)#3 (2) {
  ["email"]=>
  string(15) "ivan@server.com"
  ["name"]=>
  string(0) ""
}

Case 3:

$testName = "ivan@server.com";

Result:

object(stdClass)#3 (2) {
  ["email"]=>
  string(15) "ivan@server.com"
  ["name"]=>
  string(0) ""
}

Fetching mail from a POP3 server with Zend_Mail

Ivan Villareal — Thu, 10 Jun 2010 01:09:05 +0000

I was in a need for a class to get mails from a pop server to process them, so I’ve decided to leverage the Zend_Mail class, what I did was to create a wrapper around the Zend_Mail, to return all the messages from the server in a manageable way.

The first step was to put the needed Zend files in my library folder:


lib/Zend/Mail.php
Mail
|-- Exception.php
|-- Message
|   |-- File.php
|   `-- Interface.php
|-- Message.php
|-- Part
|   |-- File.php
|   `-- Interface.php
|-- Part.php
|-- Protocol
|   |-- Abstract.php
|   |-- Exception.php
|   |-- Imap.php
|   |-- Pop3.php
|   |-- Smtp
|   |   `-- Auth
|   |       |-- Crammd5.php
|   |       |-- Login.php
|   |       `-- Plain.php
|   `-- Smtp.php
|-- Storage
|   |-- Abstract.php
|   |-- Exception.php
|   |-- Folder
|   |   |-- Interface.php
|   |   |-- Maildir.php
|   |   `-- Mbox.php
|   |-- Folder.php
|   |-- Imap.php
|   |-- Maildir.php
|   |-- Mbox.php
|   |-- Pop3.php
|   `-- Writable
|       |-- Interface.php
|       `-- Maildir.php
|-- Storage.php
`-- Transport
    |-- Abstract.php
    |-- Exception.php
    |-- Sendmail.php
    `-- Smtp.php

Then I started writing my class:

< ?php
 
set_include_path('lib' . PATH_SEPARATOR . get_include_path());
 
include_once 'Zend/Mail.php';
include_once 'Zend/Mail/Storage/Pop3.php';
include_once 'Zend/Mail/Transport/Smtp.php';
include_once 'Zend/Mime/Part.php';
 
/**
 * Class to get mails from a server
 */
class mailFetcher
{
    private $_emailHost = 'pop.server.com';
    private $_emailUser = 'ivan@ivanvillareal.com';
    private $_emailPass = 'password';
	/**
	 * Zend_Mail_Storage
	 */
    private $_mail;
	/**
	 * Inbox wrapper
	 */
	private $_inbox = array();
 
	/**
	 * on class instantiation we fire connect
	 * to the pop server and start fetching the mails
	 */
    public function __construct()
    {
        try {
            $this->_mail = new Zend_Mail_Storage_Pop3(array(
                                             'host' => $this->_emailHost,
                                             'user' => $this->_emailUser,
                                             'password' => $this->_emailPass));
        } catch (Exception $e) {
            echo 'Mail Retrieval Failed...' . PHP_EOL;
            die();
        }      
        $this->processMail();
    }
 
 
	private function _contentDecoder($encoding, $content)
	{
		switch ($encoding) {
			case 'quoted-printable':
				$result = quoted_printable_decode($content);
			break;
			case 'base64':
				$result = base64_decode($content);
			break;
			default:
				$result = $content;
			break;
		}
		return $result;
	}
 
	/**
	 * This is where everything is done
	 */
	public function processMail()
	{
		if ($this->_mail instanceof Zend_Mail_Storage_Pop3 && $this->_mail->countMessages() > 0) {
			$messageNum = 0;
			foreach ($this->_mail as $message) {
				$messageNum++;
				$rawMail      = $this->_mail->getRawHeader($messageNum) . $this->_mail->getRawContent($messageNum);
				$received     = $message->received;
				$date         = date('Y-m-d H:i:s', strtotime($message->date));
				$messageId    = $message->headerExists('message-id') ? $message->getHeader('message-id') : null;
				//remove <> from message id
				$messageId    = (preg_match('|< (.*?)>|', $messageId, $regs)) ? $regs[1] : $messageId;
				$inReplyTo    = $message->headerExists('in-reply-to') ? $message->getHeader('in-reply-to') : null;
				//remove <> from in-reply-to
				$inReplyTo    = (preg_match('|< (.*?)>|', $inReplyTo, $regs)) ? $regs[1] : $inReplyTo;
				$references   = $message->headerExists('References')  ? $message->getHeader('References') : null;
				//remove <> from references
				$references   = (preg_match('|< (.*?)>|', $references, $regs)) ? $regs[1] : $references;
				$from         = $message->from;
				$subject      = $message->subject;
				$to           = $message->to;
				$plainContent = null;
				$htmlContent  = null;
				$attachments = array();
 
				/**
				 * Check if the message has multiple parts
				 */
		        if ($message->isMultipart()) {
					/**
					 * We have a multipart message
					 * lets extract the plain content,
					 * html content, and attachments
					 */
					foreach (new RecursiveIteratorIterator($message) as $part) {
						try {
							switch (strtok($part->contentType, ';')) {
								case 'text/plain':
									$plainContent = $this->_contentDecoder($part->getHeader('content-transfer-encoding'), $part->getContent());
								break;
								case 'text/html':
									$htmlContent = $this->_contentDecoder($part->getHeader('content-transfer-encoding'), $part->getContent());
								break;
								default: //attachment handle
									$type       = strtok($part->contentType, ';');
									$fileName   = $part->getHeader('content-description');
									$attachment = $this->_contentDecoder($part->getHeader('content-transfer-encoding'), $part->getContent());
									$attachments[] = array(
													   'file_name' => $fileName,
													   'type'      => $type,
													   'content'	=> $attachment);
								break;
							}
						} catch (Zend_Mail_Exception $e) {
							echo "$e \n";
						}
					}
				} else {
					$plainContent = $message->getContent();
				}
				$this->_inbox[] = array('messageNumber' => $messageNum,
									    'rawMail'       => $rawMail,
										'received'		=> $received,
										'date'			=> $date,
										'messageId'		=> $messageId,
										'inReplyTo'		=> $inReplyTo,
										'references'	=> $references,
										'from'			=> $from,
										'subject'		=> $subject,
										'to'			=> $to,
										'plainContent'	=> $plainContent,
										'htmlContent'	=> $htmlContent,
										'attachments'	=> $attachments);
			}
		} else {
			$this->_inbox = false;
		}
	}
	public function getInbox()
	{
		return $this->_inbox;
	}
}

My fetch class does everything on instantiation, and it will have all the mails retreived, so for example if I need to get all messages I can do this:

< ?php
include_once 'mailFetcher.php';
 
$inbox = new mailFetcher();
print_r($inbox->getInbox());

And this would be the result:

Array
(
    [0] => Array
        (
            [messageNumber] => 1
            [rawMail] => Received: from gate55.gate.iad.mailserver.com (11.1.111.1) by
 IAD2HUB01.mex02.mailserver.com (111.11.11.65) with Microsoft SMTP Server (TLS) id
 8.2.247.2; Wed, 9 Jun 2010 17:56:16 -0400
Received: from mail-yw0-f186.google.com (mail-yw0-f186.google.com
 [111.11.111.186])	by gate55.gate.iad.mailserver.com (SMTP Server) with ESMTP id
 B1F6E31D0171	for ; Wed,  9 Jun 2010
 17:56:15 -0400 (EDT)
Received: by ywh16 with SMTP id 16so5309053ywh.32        for
 ; Wed, 09 Jun 2010 14:56:15 -0700 (PDT)
Received: by 10.111.111.111 with SMTP id jz1mr465966qcb.170.1276120573865;
 	Wed, 09 Jun 2010 14:56:13 -0700 (PDT)
Received: by 11.111.111.11 with HTTP; Wed, 9 Jun 2010 14:56:13 -0700 (PDT)
From: Ivan Villareal 
To: "asdfasdf@ivanvillareal.com" 
Date: Wed, 9 Jun 2010 17:56:13 -0400
Subject: attachment test
Thread-Topic: attachment test
Thread-Index: AcsIHpXuE+vWJJMfTGyqikgD2kYkng==
Message-ID: 
Accept-Language: en-US
Content-Language: en-US
X-MS-Exchange-Organization-AuthAs: Anonymous
X-MS-Exchange-Organization-AuthSource: IAD2HUB01.mex02.mailserver.com
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
x-virus-scanned: OK, catchall
x-originating-ip: [111.11.111.186]
dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed;        d=gmail.com;
 s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=IvMOHmKRabcJ2PrUHPHB/NbNpgQubvvgGprC6gtDY9E=;
        b=cA5Pjycedt9Pylic7CTk4HCF6HXV9eEFpr086KwcMepGKJEv4DWU6d4iFt4Gp7R42V
         +4rVmrmwj2j9yj1vbEUqVMI6/m+Kkwt1r+MAtN/0qZsvEaVBBbbjTBxFLO9cQ7xlaSSm
         eGTHv9mR0qCcBZGLOVFeWUNiYDkhY2F0eDUbU=
domainkey-signature: a=rsa-sha1; c=nofws;        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=VKdlRSfBz/Swij0xE0c7Su8TOwcc2YGmw0K8dvVxoS9TJf0p8nIqtR6nQtmKqabSu4
         g2ALFFHXuVOhGpP+JdvI3wQJ1k9N5EZhcWtSRECrikOlLnaDA8/4z3SYckY6hs7Q+KIe
         qG0IbY+QkuZh7yymcyAN5omTaFDFT2GIPs+1c=
x-orig-to: catchall@ivanvillareal.com
Content-Type: multipart/mixed;
	boundary="_004_AANLkTingkMPbKeFog5kxXoq8WRib7tJ1pT1Rsy1ug1mailgmailcom_"
MIME-Version: 1.0
 
--_004_AANLkTingkMPbKeFog5kxXoq8WRib7tJ1pT1Rsy1ug1mailgmailcom_
Content-Type: multipart/alternative;
	boundary="_000_AANLkTingkMPbKeFog5kxXoq8WRib7tJ1pT1Rsy1ug1mailgmailcom_"
 
--_000_AANLkTingkMPbKeFog5kxXoq8WRib7tJ1pT1Rsy1ug1mailgmailcom_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
 
amhnamhnamhnDQo=
 
--_000_AANLkTingkMPbKeFog5kxXoq8WRib7tJ1pT1Rsy1ug1mailgmailcom_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64
 
amhnamhnamhnDQo=
 
--_000_AANLkTingkMPbKeFog5kxXoq8WRib7tJ1pT1Rsy1ug1mailgmailcom_--
 
--_004_AANLkTingkMPbKeFog5kxXoq8WRib7tJ1pT1Rsy1ug1mailgmailcom_
Content-Type: image/jpeg; name="2010-05-28-112655.jpg"
Content-Description: 2010-05-28-112655.jpg
Content-Disposition: attachment; filename="2010-05-28-112655.jpg"; size=15004;
	creation-date="Wed, 09 Jun 2010 17:56:16 GMT";
	modification-date="Wed, 09 Jun 2010 17:56:16 GMT"
Content-Transfer-Encoding: base64
 
/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0a
gq4UlFJmmKwtHakzS0DsMNFLikIxQAUmKKM0AFLSUtABRRRQAlFGeaKACiikouAtGaKPrQAmaKDS
4oASgClxRQKwUUUlAxaKKM0AFBNFGM0AGalHzIKiIp6HgigQuKKKKQyZDlaJAcU2I09+VP0qWSRK
R0zTGXBpuec1IDvX3qihqNhhV5WG0Vn9DUocgUMTJpyCpIqsKezZXmoqENDqQ0dqBmmMBS0UmKAF
oFJS0AJ3oIpQKQk0AAp4JxTBTxQwYE0oFNNOU8UhCkDGKaRin0EZoEiLIpcUuzuKMUyrje9FKQet
J2oGf//Z
 
--_004_AANLkTingkMPbKeFog5kxXoq8WRib7tJ1pT1Rsy1ug1mailgmailcom_--
 
            [received] => from gate55.gate.iad.mailserver.com (11.1.111.1) by IAD2HUB01.mex02.mailserver.com (111.11.11.15) with Microsoft SMTP Server (TLS) id 1.1.147.2; Wed, 9 Jun 2010 17:56:16 -0400
from mail-yw0-f186.google.com (mail-yw0-f186.google.com [209.85.211.186])	by gate55.gate.iad.mailserver.com (SMTP Server) with ESMTP id B1F6E31D0171	for ; Wed,  9 Jun 2010 17:56:15 -0400 (EDT)
by ywh16 with SMTP id 16so5309053ywh.32        for ; Wed, 09 Jun 2010 14:56:15 -0700 (PDT)
by 10.229.233.193 with SMTP id jz1mr465966qcb.170.1276120573865; Wed, 09 Jun 2010 14:56:13 -0700 (PDT)
by 10.229.192.82 with HTTP; Wed, 9 Jun 2010 14:56:13 -0700 (PDT)
            [date] => 2010-06-09 14:56:13
            [messageId] => AANLkTingkMPbKeFog5kxXoq8WRib7tJ1pT1Rsy-1ug-1@mail.gmail.com
            [inReplyTo] => 
            [references] => 
            [from] => Ivan Villareal 
            [subject] => attachment test
            [to] => "asdfasdf@ivanvillareal.com" 
            [plainContent] => jhgjhgjhg
 
            [htmlContent] => jhgjhgjhg
 
            [attachments] => Array
                (
                    [0] => Array
                        (
                            [file_name] => 2010-05-28-112655.jpg
                            [type] => image/jpeg
                        )
 
                )
 
        )
 
)

From this point I can iterate the inbox easily saving the email results or applying other logic to it,

Apache virtual host script in perl

Ivan Villareal — Thu, 27 May 2010 23:57:26 +0000

This is a very short post about a little script I did to create remove virtual hosts easily.

I’ve found a couple of scripts (http://patrickgibson.com/utilities/virtualhost/, http://www.tamdenholm.com/portfolio/apache) to do this but they didn’t worked right, I’m using Ubuntu Karmic, maybe thats why. However instead of fixing those scripts I went to my vim editor and started typing this, I’ve spent about 2 hours writing this while I was watching a crappy tv program, but in the end it worked, it surely contains some bugs, or it is unsafe to run it, I just wanted to share this.

#!/usr/bin/perl -w
use strict;
use File::Path qw(mkpath rmtree);
use Getopt::Long;
 
 
our $ipAddress        = '127.0.0.1';
our $apacheConfigFile = 'apache2.conf';
our $apacheConfigDir  = '/etc/apache2';
our $sitesAvailable   = 'sites-available';
 
our $docRootPrefix = '/home/ivan/www';
our $docRoot = 'public_html';
our $logsDir  = 'logs';
 
 
my $del = '';
my $add = '';
my $domain = '';
 
unless (GetOptions ('del' => \$del, 'add' => \$add, 'domain=s' => \$domain) or usage()) {
    usage();
}
 
#print $paramResults;
if ($add || $del) {
    if ($domain) {
        if ($add) {
            createVhost($domain);
        } elsif ($del) {
            deleteVhost($domain);
        }
    } else {
        usage();
    }
   #print 'add aqui';
} else {
    usage();
}
 
sub usage {
    print < <USAGE
This program will add or remove
apache virtual hosts on ubuntu karmic 9.10
 
usage: vhost-updater.pl [--add | --del] --domain newhost.tld 
USAGE
}
 
sub returnVhostPaths
{
    my $vhost = shift;
    my @dir = split(/\//, $docRootPrefix);
    my %res;
 
    push(@dir, $vhost);
 
    my $hostDir = join('/', @dir);
    $res{'docRoot'} = $hostDir . '/' . $docRoot;
    $res{'logsDir'} = $hostDir . '/' . $logsDir;
 
    #todo dir validation
    @dir = split(/\//, $apacheConfigDir);
    push(@dir, $sitesAvailable);
    push(@dir, $vhost);
    $res{'apacheConfig'} = join('/', @dir);
 
    return %res;   
}
 
sub createVhost {
    my $vhost = shift;
    #first create the docRoot
    my %vhostInfo = returnVhostPaths($vhost);
 
    informOut("Creating docroot dir: $vhostInfo{'docRoot'}");
    informOut("Creating log dir: $vhostInfo{'logsDir'}");
    mkpath($vhostInfo{'docRoot'}, $vhostInfo{'logsDir'});
 
 
    informOut("Site File: $vhostInfo{'apacheConfig'}");
 
    my $vhostContent = << "EOF";

    ServerName $vhost
    DocumentRoot $vhostInfo{'docRoot'}
    
        Options Indexes FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
    
        ErrorLog $vhostInfo{'logsDir'}/error_log
        CustomLog $vhostInfo{'logsDir'}/access_log "%h %l %u %t \\"%r\\" %>s %b \\"%{Referer}i\\" \\"%{User-agent}i\\""
        LogLevel debug
 
 
EOF
    informOut("Creating Vhost...");
    open FILE, ">", $vhostInfo{'apacheConfig'} or die $!;
    print FILE $vhostContent;
    close FILE;
 
    informOut("Adding host $vhost");
    open FILE, ">>", '/etc/hosts' or die $!;
    print FILE $ipAddress ."\t". $vhost ."\n";
    close FILE;
 
    my $output = `/usr/sbin/a2ensite $vhost`;
    print $output;
 
    restartApache();
    #print $vhostConten t;
}
 
sub restartApache
{
    informOut("Restarting apache");
    my $output = `/etc/init.d/apache2 restart`;
    print $output; 
}
 
sub deleteVhost
{
    my $vhost = shift;
     my %vhostInfo = returnVhostPaths($vhost);
 
    informOut("Removing $vhost from hosts file");
    open IN, '< ', '/etc/hosts' or die $!;
    my @hostsFile = ;
    close IN;
 
    my @contents = grep(!/^127.0.0.1\t$vhost/, @hostsFile);
 
    open FILE, ">", '/etc/hosts' or die $!;
    print FILE @contents;
    close FILE;
 
    my $output = `/usr/sbin/a2dissite $vhost`;
    print $output;
 
    informOut("Removing  $vhostInfo{'apacheConfig'} file");
    unlink($vhostInfo{'apacheConfig'});
 
    restartApache();
 
    print " manually remove $vhostInfo{'docRoot'}... \n";
 
}
 
sub informOut {
    my $message = shift;
    print "$message \n";
}

Here is a screenshot of the program running

Bulk domain name resolver with perl

Ivan Villareal — Fri, 07 May 2010 10:00:00 +0000

I was in the need of getting the ip address from a long list of domain names, the domains were stored in a MySQL db, so I quickly turned into perl for making a script that got the list and then use an external command to resolve the domain.

At first I was going to use the dig command, but that would have taken to much effort, then I thought about using the ping command, but again to much effort, instead I’ve decided to leverage the IO::Socket::INET module, trying to open a socket to a domain name, will resolve the name and I could use that.

My implementation was very simple and it worked well, here is the relevant part:

#include the module
use IO::Socket::INET;
 
#create a socket to uncommon udp port, because tcp is likely to be firewalled
my $sock = IO::Socket::INET->new(
                           PeerAddr=> "$domain",
                           PeerPort =>40125,
                           Proto=>'udp');
 
#if we can create a socket, then we have an ip
if (defined $sock) {
         $result = $sock->peerhost;
         logOut("Success!");
    } else {
        logOut("Failed trying to connect to $domain");
        return;
    }

This was Pretty simple no?, here is the complete script, which gathers all the rows with no ip address and then it will try to get the ip, update the row and continue.

#!/usr/bin/perl
use strict;
use DBI;
use IO::Socket::INET;
 
####Global Variables###
 
our %db = ('host' => 'localhost','name' => 'domains','user' => 'ivan','pass' => '1234','port' => '3306');
our $dsn;
our $dbh;
 
#log function
sub logOut {
    my ($msg) = @_;
    print $msg . "\n";
}
 
#db connection
sub initialize {
    logOut("Connecting to db server on $db{'host'}");
    $dsn = "DBI:mysql:database=$db{'name'};host=$db{'host'};port=$db{'port'}";
    $dbh = DBI->connect($dsn, $db{'user'}, $db{'pass'}) or die('failed to connect');
}
 
#function to gather the ip fo a domain
sub getIpAddress {
    my ($domain) = @_;
    logOut("Trying to connect to: $domain...");
    my $result;
    my $sock = IO::Socket::INET->new(
                           PeerAddr=> "$domain",
                           PeerPort =>40125,
                           Proto=>'udp');
    if (defined $sock) {
         $result = $sock->peerhost;
         logOut("Success!");
    } else {
        logOut("Failed trying to connect to $domain");
        return;
    }
    return $result;
}
 
##This function
# gets all domains from domains table
sub getDomains {
    logOut("Fetching db for domains...");
    my $result = $dbh->selectall_hashref("SELECT domain_name FROM domains WHERE ip_address IS NULL", 'domain_name');
    logOut(keys( %$result ) . " domains returned.");
    my $updatedDomains = 0;
    my $noIpDomains    = 0;
    foreach my $id (keys %$result) {
        my $ip = getIpAddress($result->{$id}->{domain_name});
        if ($ip) {
            logOut("Ip found $ip for $result->{$id}->{domain_name}");
            my @values = ($ip, $result->{$id}->{domain_name});
            my $re = $dbh->do("UPDATE domains SET ip_address=? WHERE domain_name=?",undef, @values);
                if ($re == 1) {
                    $updatedDomains++;
                }
        } else {
            logOut("no ip for $result->{$id}->{domain_name}");
            $noIpDomains++;
        }
    }
    logOut("Updated Domains: $updatedDomains");
    logOut("Domains with no Ip found: $noIpDomains");
}
 
#script starts here
initialize();
getDomains();

Maybe there are more efficient methods but as I said, this worked pretty well, it didn’t consume a lot of results and it performs fast enough for my needs.

Partial Db backups with mysqldump

Ivan Villareal — Wed, 05 May 2010 08:30:09 +0000

I was updating my db dev instance with a set of changes, and when I was restoring the dump for a table it contained a drop db clause it was a simple table and it took a about 15 seconds to finish the operation, when I looked at the db schema all the tables were gone!

This db has a very large table about 16Gb of binary data, so getting a new dump would require a lot of time, so as workaround I’ve dumped only sets of data that I need to my tests.

First I did a dump of the db schema with no data:

mysqldump -h host-u ${USER} --password=${PASS} --no-data --add-drop-database --databases largeDb> schema_largeDb.sql

then I created a dump for the tables that I need, for this I have to include a list of tables that I don’t want, I’ve created the list of unwanted tables manually.

mysqldump -h host-u ${USER} --password=${PASS} --no-create-info --ignore-table=largeDb.unwantedTable1 --ignore-table=largeDb.unwantedTable2 --ignore-table=largeDb.unwantedTable3 > smallData.sql

And because I need data from the big table, I’ve used the –where clause to limit the rows returned in

mysqldump -h host-u ${USER} --password=${PASS} "--where=brand_id='2496'" largeDb bigTable > binaryData.sql

With this one I was able to get only the binaryData that I need to make my tests, instead of waiting to get for the entire dbDump,

Upgrading php5.1 to php5.3 on CENTOS 4

Ivan Villareal — Thu, 22 Apr 2010 20:23:34 +0000

I needed to update the php version of a centos 4 box, so this is what I did:

I’ve decided to go for the easy way, using the remi repo, so first I’ve Downloaded the Extra Packages for Enterprise Linux (EPEL)

[root@box php5.3]# wget http://download.fedora.redhat.com/pub/epel/4/i386/epel-release-4-9.noarch.rpm
--13:08:07--  http://download.fedora.redhat.com/pub/epel/4/i386/epel-release-4-9.noarch.rpm
           => `epel-release-4-9.noarch.rpm'
Resolving download.fedora.redhat.com... 209.132.183.67
Connecting to download.fedora.redhat.com|209.132.183.67|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13,686 (13K) [application/x-rpm]
 
100%[=================================================================================>] 13,686        --.--K/s            
 
13:08:08 (405.60 KB/s) - `epel-release-4-9.noarch.rpm' saved [13686/13686]

Then the remi repo:

[root@box php5.3]# wget http://rpms.famillecollet.com/enterprise/remi-release-4.rpm
--13:08:21--  http://rpms.famillecollet.com/enterprise/remi-release-4.rpm
           => `remi-release-4.rpm'
Resolving rpms.famillecollet.com... 88.191.60.189
Connecting to rpms.famillecollet.com|88.191.60.189|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5,120 (5.0K) [application/x-rpm]
 
100%[=================================================================================>] 5,120         32.45K/s            
 
13:08:22 (32.41 KB/s) - `remi-release-4.rpm' saved [5120/5120]

Then I’ve upgraded both repos:

[root@box php5.3]# rpm -Uvh remi-release-4*.rpm epel-release-4*.rpm
warning: remi-release-4.rpm: V3 DSA signature: NOKEY, key ID 00f97f56
warning: epel-release-4-9.noarch.rpm: V3 DSA signature: NOKEY, key ID 217521f6
Preparing...                ########################################### [100%]
   1:epel-release           ########################################### [ 50%]
   2:remi-release           ########################################### [100%]

and after that I downloaded the remi repo file for Yum:

[root@box php5.3]# cd /etc/yum.repos.d/
[root@box yum.repos.d]# wget http://rpms.famillecollet.com/remi-enterprise.repo
--13:09:28--  http://rpms.famillecollet.com/remi-enterprise.repo
           => `remi-enterprise.repo'
Resolving rpms.famillecollet.com... 88.191.60.189
Connecting to rpms.famillecollet.com|88.191.60.189|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 793 [text/plain]
 
100%[=================================================================================>] 793           --.--K/s            
 
13:09:28 (50.42 MB/s) - `remi-enterprise.repo' saved [793/793]

After that I just told yum to upgrade php:

[root@box yum.repos.d]# yum --enable remi update php
Repository remi is listed more than once in the configuration
Repository remi-test is listed more than once in the configuration
Setting up Update Process
Setting up repositories
epel                      100% |=========================| 3.4 kB    00:00    
remi                      100% |=========================| 2.6 kB    00:00    
Reading repository metadata in from local files
357f538e0bf37d10983cd3b22 100% |=========================| 732 kB    00:06    
epel      : ################################################## 2267/2267
e849a85ae6e8f1b8e5e71c08b 100% |=========================| 128 kB    00:03    
remi      : ################################################## 444/444
Resolving Dependencies
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for php to pack into transaction set.
php-5.3.2-1.el4.remi.x86_ 100% |=========================|  19 kB    00:00    
---> Package php.x86_64 0:5.3.2-1.el4.remi set to be updated
--> Running transaction check
--> Processing Dependency: php-common = 5.3.2-1.el4.remi for package: php
--> Processing Dependency: php-cli = 5.3.2-1.el4.remi for package: php
--> Restarting Dependency Resolution with new changes.
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for php-common to pack into transaction set.
php-common-5.3.2-1.el4.re 100% |=========================|  25 kB    00:01    
---> Package php-common.x86_64 0:5.3.2-1.el4.remi set to be updated
---> Downloading header for php-cli to pack into transaction set.
php-cli-5.3.2-1.el4.remi. 100% |=========================|  19 kB    00:01    
---> Package php-cli.x86_64 0:5.3.2-1.el4.remi set to be updated
--> Running transaction check
--> Processing Dependency: php-common = 5.1.6-3.el4s1.10 for package: php-mysql
--> Processing Dependency: php-common = 5.1.6-3.el4s1.10 for package: php-xml
--> Processing Dependency: php-common = 5.1.6-3.el4s1.10 for package: php-pdo
--> Processing Dependency: libedit.so.0()(64bit) for package: php-cli
--> Restarting Dependency Resolution with new changes.
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for php-xml to pack into transaction set.
php-xml-5.3.2-1.el4.remi. 100% |=========================|  20 kB    00:00    
---> Package php-xml.x86_64 0:5.3.2-1.el4.remi set to be updated
---> Downloading header for libedit to pack into transaction set.
libedit-20090923-3.0_1.el 100% |=========================| 5.8 kB    00:00    
---> Package libedit.x86_64 0:20090923-3.0_1.el4.rf set to be updated
---> Downloading header for php-pdo to pack into transaction set.
php-pdo-5.3.2-1.el4.remi. 100% |=========================|  19 kB    00:00    
---> Package php-pdo.x86_64 0:5.3.2-1.el4.remi set to be updated
---> Downloading header for php-mysql to pack into transaction set.
php-mysql-5.3.2-1.el4.rem 100% |=========================|  19 kB    00:00    
---> Package php-mysql.x86_64 0:5.3.2-1.el4.remi set to be updated
--> Running transaction check
--> Processing Dependency: libsqlite.so.0()(64bit) for package: php-pdo
--> Processing Dependency: libmysqlclient.so.16(libmysqlclient_16)(64bit) for package: php-mysql
--> Processing Dependency: libmysqlclient.so.16()(64bit) for package: php-mysql
--> Restarting Dependency Resolution with new changes.
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for sqlite2 to pack into transaction set.
sqlite2-2.8.17-2.el4.remi 100% |=========================| 5.2 kB    00:00    
---> Package sqlite2.x86_64 0:2.8.17-2.el4.remi set to be updated
---> Downloading header for mysql-libs to pack into transaction set.
mysql-libs-5.1.45-1.el4.r 100% |=========================|  24 kB    00:00    
---> Package mysql-libs.x86_64 0:5.1.45-1.el4.remi set to be updated
--> Running transaction check
--> Processing Dependency: libmysqlclient.so.15()(64bit) for package: perl-DBD-MySQL
--> Processing Dependency: libmysqlclient.so.15(libmysqlclient_15)(64bit) for package: mysql-server
--> Processing Dependency: libmysqlclient_r.so.15(libmysqlclient_15)(64bit) for package: mysql-server
--> Processing Dependency: libmysqlclient.so.15()(64bit) for package: mysql-server
--> Processing Dependency: libmysqlclient.so.15(libmysqlclient_15)(64bit) for package: perl-DBD-MySQL
--> Processing Dependency: mysql-libs = 5.0.82sp1-1.el4_8 for package: mysql
--> Processing Dependency: libmysqlclient.so.15(libmysqlclient_15)(64bit) for package: mysql
--> Processing Dependency: libmysqlclient.so.15()(64bit) for package: mysql
--> Processing Dependency: libmysqlclient_r.so.15()(64bit) for package: mysql-server
--> Restarting Dependency Resolution with new changes.
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for mysqlclient15 to pack into transaction set.
mysqlclient15-5.0.67-1.el 100% |=========================| 3.9 kB    00:00    
---> Package mysqlclient15.x86_64 0:5.0.67-1.el4.remi set to be updated
---> Package perl-DBD-mysql.x86_64 0:4.013-1.el4.rf set to be updated
---> Downloading header for mysql to pack into transaction set.
mysql-5.1.45-1.el4.remi.x 100% |=========================|  19 kB    00:00    
---> Package mysql.x86_64 0:5.1.45-1.el4.remi set to be updated
---> Downloading header for mysql-server to pack into transaction set.
mysql-server-5.1.45-1.el4 100% |=========================|  27 kB    00:00    
---> Package mysql-server.x86_64 0:5.1.45-1.el4.remi set to be updated
--> Running transaction check
--> Processing Dependency: libmysqlclient.so.14(libmysqlclient_14)(64bit) for package: perl-DBD-mysql
--> Processing Dependency: libmysqlclient.so.14()(64bit) for package: perl-DBD-mysql
--> Restarting Dependency Resolution with new changes.
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for mysqlclient14 to pack into transaction set.
mysqlclient14-4.1.22-1.el 100% |=========================| 5.5 kB    00:00    
---> Package mysqlclient14.x86_64 0:4.1.22-1.el4.remi set to be updated
--> Running transaction check
 
Dependencies Resolved
 
=============================================================================
Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
perl-DBD-mysql          x86_64     4.013-1.el4.rf   rpmforge          228 k
     replacing  perl-DBD-MySQL.x86_64 4.008-1.el4s1
 
Updating:
php                     x86_64     5.3.2-1.el4.remi  remi              1.4 M
Installing for dependencies:
libedit                 x86_64     20090923-3.0_1.el4.rf  rpmforge          265 k
mysqlclient14           x86_64     4.1.22-1.el4.remi  remi              1.2 M
mysqlclient15           x86_64     5.0.67-1.el4.remi  remi              1.3 M
sqlite2                 x86_64     2.8.17-2.el4.remi  remi              161 k
Updating for dependencies:
mysql                   x86_64     5.1.45-1.el4.remi  remi              1.0 M
mysql-libs              x86_64     5.1.45-1.el4.remi  remi              1.7 M
mysql-server            x86_64     5.1.45-1.el4.remi  remi               11 M
php-cli                 x86_64     5.3.2-1.el4.remi  remi              2.6 M
php-common              x86_64     5.3.2-1.el4.remi  remi              963 k
php-mysql               x86_64     5.3.2-1.el4.remi  remi              155 k
php-pdo                 x86_64     5.3.2-1.el4.remi  remi              177 k
php-xml                 x86_64     5.3.2-1.el4.remi  remi              232 k
 
Transaction Summary
=============================================================================
Install      5 Package(s)        
Update       9 Package(s)        
Remove       0 Package(s)        
Total download size: 22 M
Is this ok [y/N]: y
Downloading Packages:
(1/14): php-xml-5.3.2-1.e 100% |=========================| 232 kB    00:04    
(2/14): libedit-20090923- 100% |=========================| 265 kB    00:04    
(3/14): php-common-5.3.2- 100% |=========================| 963 kB    00:15    
(4/14): mysqlclient15-5.0 100% |=========================| 1.3 MB    00:23    
(5/14): php-cli-5.3.2-1.e 100% |=========================| 2.6 MB    00:35    
(6/14): perl-DBD-mysql-4. 100% |=========================| 228 kB    00:04    
(7/14): php-pdo-5.3.2-1.e 100% |=========================| 177 kB    00:05    
(8/14): mysql-5.1.45-1.el 100% |=========================| 1.0 MB    00:15    
(9/14): sqlite2-2.8.17-2. 100% |=========================| 161 kB    00:05    
(10/14): mysqlclient14-4. 100% |=========================| 1.2 MB    00:15    
(11/14): mysql-server-5.1 100% |=========================|  11 MB    01:59    
(12/14): php-5.3.2-1.el4. 100% |=========================| 1.4 MB    00:23    
(13/14): mysql-libs-5.1.4 100% |=========================| 1.7 MB    00:23    
(14/14): php-mysql-5.3.2- 100% |=========================| 155 kB    00:04    
warning: rpmts_HdrFromFdno: V3 DSA signature: NOKEY, key ID 00f97f56
Public key for php-xml-5.3.2-1.el4.remi.x86_64.rpm is not installed
Retrieving GPG key from http://rpms.famillecollet.com/RPM-GPG-KEY-remi
Importing GPG key 0x00F97F56 "Remi Collet "
Is this ok [y/N]:
 
Key imported successfully
warning: only V3 signatures can be verified, skipping V4 signature
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
 
WARNING : This php-* RPM are not official Fedora build and
overrides the official ones. Don't file bugs on Fedora Project.
 
Use dedicated forums http://forums.famillecollet.com/
 
  Updating  : php-common                                           [ 1/24]warning: /etc/php.ini created as /etc/php.ini.rpmnew
  Updating  : php-common                   ####################### [ 1/24]
  Installing: mysqlclient15                ####################### [ 2/24]
warning: only V3 signatures can be verified, skipping V4 signature
 
WARNING : This MySQL RPM is not an official Fedora build and it
overrides the official one. Don't file bugs on Fedora Project.
Use dedicated forums http://forums.famillecollet.com/
 
  Updating  : mysql-libs                                           [ 3/24]warning: /etc/my.cnf created as /etc/my.cnf.rpmnew
  Updating  : mysql-libs                   ####################### [ 3/24]
  Updating  : mysql                        ####################### [ 4/24]
  Installing: mysqlclient14                ####################### [ 5/24]
  Installing: sqlite2                      ####################### [ 6/24]
  Updating  : php-pdo                      ####################### [ 7/24]
  Installing: libedit                      ####################### [ 8/24]
  Updating  : php-cli                      ####################### [ 9/24]
  Updating  : php                          ####################### [10/24]
  Updating  : php-mysql                    ####################### [11/24]
  Updating  : php-xml                      ####################### [12/24]
  Installing: perl-DBD-mysql               ####################### [13/24]
  Updating  : mysql-server                 ####################### [14/24]
  Cleanup   : php-xml                      ####################### [15/24]
  Cleanup   : php-common                   ####################### [16/24]
  Cleanup   : php-cli                      ####################### [17/24]
  Removing  : perl-DBD-MySQL               ####################### [18/24]
  Cleanup   : php-pdo                      ####################### [19/24]
  Cleanup   : mysql                        ####################### [20/24]
  Cleanup   : mysql-server                 ####################### [21/24]
  Cleanup   : php                          ####################### [22/24]
  Cleanup   : mysql-libs                   ####################### [23/24]
  Cleanup   : php-mysql                    ####################### [24/24]
 
Installed: perl-DBD-mysql.x86_64 0:4.013-1.el4.rf
Dependency Installed: libedit.x86_64 0:20090923-3.0_1.el4.rf mysqlclient14.x86_64 0:4.1.22-1.el4.remi mysqlclient15.x86_64 0:5.0.67-1.el4.remi sqlite2.x86_64 0:2.8.17-2.el4.remi
Updated: php.x86_64 0:5.3.2-1.el4.remi
Dependency Updated: mysql.x86_64 0:5.1.45-1.el4.remi mysql-libs.x86_64 0:5.1.45-1.el4.remi mysql-server.x86_64 0:5.1.45-1.el4.remi php-cli.x86_64 0:5.3.2-1.el4.remi php-common.x86_64 0:5.3.2-1.el4.remi php-mysql.x86_64 0:5.3.2-1.el4.remi php-pdo.x86_64 0:5.3.2-1.el4.remi php-xml.x86_64 0:5.3.2-1.el4.remi
Replaced: perl-DBD-MySQL.x86_64 0:4.008-1.el4s1
Complete!

When this was done, I just synced /etc/php.ini with the new one /etc/php.ini.rpmnew and /etc/my.cnf with /etc/my.cnf.rpmnew when this was done I just restarted mysql and apache and I php5.3 was now available.

GPS tracking fleet project (part. 1)

Ivan Villareal — Mon, 22 Mar 2010 02:59:24 +0000

About a year ago I was talking to a friend that was working for a company that offered a fleet tracking service, he described the process and I just got curious about how could I implement the entire system easily, I did some research and there wasn’t a lot of information about the entire picture, there were bits here and there but nothing concrete.

After a couple of days of researching I came up with a Plan to write a platform in one month in my spare time, the intention was to create a fully functional backend and front end pieces, with the ability to track any number of units.

It was a bit challenging because :

The short time span I wanted to have this ready.
I didn’t have any hardware to test (GPS receivers, GPRS modems, or even a dedicated server to put the backend), and I didn’t want to spent any money buying any of this.
I didn’t had a lot of experience with Google maps API.

The first thing I did was a diagram of how the system should work:

I’m not good with drawings so bear with me, let me explain the diagram:

There will be a tracking device in every truck that will be sending periodic updates about its location, speed, and other events trough a GPRS link, provided by a conventional GSM network, the device must be programmed to send this updates to a certain ip address (Application server), at specified periods of time.

The application server (background process) will have a daemon running on a specific port waiting for requests from tracking devices, if the tracking device is authorized it will accept the stream and save the info received from the device to a db.

The requirements for the backend application were:

It should be able to run on Linux.
The process should be able to handle a large number of requests.
It needed to be run as a daemon, interfacing with a system to logger.
It must be able to decode NMEA sentences and save them in a MySQL Db.

The front end application, requirements were:

Handle user authentication.
Present the gathered data in real time.
Generate reports.

Technology selection:

With basic requirements on hand, the next thing was to look the technologies I should be using and this were my choices:

For the backend:

Gentoo Linux
Java 1.6
Mysql 5.0

For the frontend:

Gentoo Linux
PHP 5.2
Jquery 1.3
Mysql 5.0

In my next post I will describe the approach I took on the back end process and how I started building it.

PHP cross domain ajax Proxy

Ivan Villareal — Fri, 19 Mar 2010 23:52:04 +0000

I received a javascript application, that made cross domain calls to a webservice using the flensed library, this was pretty neat, but filling the dropdown boxes was very slow. Not to mention unnecessary, because I was going to embed the dropdown values into this modal window.

What I did was to refactor the javascript app, into php code to fill the values, and left the important webservice requests alone, I didn’t want to use the flXHR library, because the widget was so simple, so what I did was a php json-rpc proxy client, winch its work would be to post the data received from the client to a remote webservice.

I’m using Zend_Json to encode/decode json, because the php version on production is 5.1, and the php json extensions are not available.

First I create an object that will hold the data to be sent.

$re = new stdClass();
$re->brand_id  = $_POST['brand_id'];
$names         = $_POST['domains'];
 
$domains = explode(",", $names);
$filteredDomains = array();
foreach ($domains as $domain) {
 $domainParts = parse_url($domain);
 if (isset($domainParts['host']) && $domainParts['host'] != '') {
 $filteredDomains[] = $domainParts['host'];
 }
}
$re->domains   = $filteredDomains;

That was simple, I didn’t have to worry about filtering the user input, because the webservice will take care of that, the only thing I do, is create an array of domains, with has only the host part.

after I have my Object ready I encode it as a json string, open a socket and make the request, here is how I did it:

$request = Zend_Json::encode($re);
$opts = array ('http' => array (
    'method'  => 'POST',
    'header'  => 'Content-type: application/json',
    'content' => $request
    ));
$context  = stream_context_create($opts);
$conection = $fp = fopen($url, 'r', false, $context);
if ($conection) {
 $reply = '';
 while($row = fgets($fp)) {
 $reply.= trim($row)."\n";
}
$reply = Zend_Json::decode($reply);

Pretty simple no?

Getting first and last row from a group set without sub queries in mysql

Ivan Villareal — Fri, 19 Feb 2010 01:45:24 +0000

When I use the GROUP BY clause in mysql It starts grouping by the first different row in the resultset so for example if I have this table:

mysql> select * from grouptest;
+----+---------+-------+------------+
| id | city    | value | date       |
+----+---------+-------+------------+
|  1 | tijuana |    10 | 2010-02-02 | 
|  2 | tijuana |    11 | 2010-02-09 | 
|  3 | tijuana |    15 | 2010-02-12 | 
|  4 | tijuana |    17 | 2010-02-17 | 
|  5 | tecate  |    12 | 2010-02-10 | 
|  6 | tecate  |     1 | 2010-02-14 | 
|  7 | tecate  |    29 | 2010-02-17 | 
+----+---------+-------+------------+
7 rows in set (0.00 sec)

If I group by city this is what I get:

mysql> SELECT * FROM grouptest g GROUP BY city;
+----+---------+-------+------------+
| id | city    | value | date       |
+----+---------+-------+------------+
|  5 | tecate  |    12 | 2010-02-10 | 
|  1 | tijuana |    10 | 2010-02-02 | 
+----+---------+-------+------------+
2 rows in set (0.00 sec)

But I want to get the last inserted rows, so could do a sub query like this:

mysql> SELECT * FROM grouptest g
    -> WHERE g.date = (SELECT MAX(date) FROM grouptest gt WHERE g.city = gt.city)
    -> GROUP BY g.city;
+----+---------+-------+------------+
| id | city    | value | date       |
+----+---------+-------+------------+
|  7 | tecate  |    29 | 2010-02-17 | 
|  4 | tijuana |    17 | 2010-02-17 | 
+----+---------+-------+------------+
2 rows in set (0.00 sec)

But if the db is large making subqueries could be expensive, so a cleaner solution might be using the GROUP_CONCAT function, so for example the above query could be written like this:

mysql> SELECT id, city, value, date, GROUP_CONCAT(date) AS all_dates
    -> FROM grouptest g
    -> GROUP BY city;
+----+---------+-------+------------+---------------------------------------------+
| id | city    | value | date       | all_dates                                   |
+----+---------+-------+------------+---------------------------------------------+
|  5 | tecate  |    12 | 2010-02-10 | 2010-02-10,2010-02-14,2010-02-17            | 
|  1 | tijuana |    10 | 2010-02-02 | 2010-02-02,2010-02-09,2010-02-12,2010-02-17 | 
+----+---------+-------+------------+---------------------------------------------+
2 rows in set (0.00 sec)

At this point I can explode the all_dates column in php or split them in perl to get the last date for each city, but a little more work on our query could do our job simpler, something like this for instance:

mysql> SELECT id, city, value, SUBSTRING_INDEX(GROUP_CONCAT(date),',',-1) AS date
    -> FROM grouptest g
    -> GROUP BY city;
+----+---------+-------+------------+
| id | city    | value | date       |
+----+---------+-------+------------+
|  5 | tecate  |    12 | 2010-02-17 | 
|  1 | tijuana |    10 | 2010-02-17 | 
+----+---------+-------+------------+
2 rows in set (0.00 sec)

Or if I need to get the first and last rows for each group set I just add the date column like this:

mysql> SELECT id, city, value, date as first_date, SUBSTRING_INDEX(GROUP_CONCAT(date),',',-1) AS last_date
    -> FROM grouptest g
    -> GROUP BY city;
+----+---------+-------+------------+------------+
| id | city    | value | first_date | last_date  |
+----+---------+-------+------------+------------+
|  5 | tecate  |    12 | 2010-02-10 | 2010-02-17 | 
|  1 | tijuana |    10 | 2010-02-02 | 2010-02-17 | 
+----+---------+-------+------------+------------+
2 rows in set (0.00 sec)

There may be several other ways around this, but for now this approach is enough, I’m not sure about the impact this 2 functions may have on the server but I’m sure it is less than executing a subquery.

Compressing/Decompressing mysql dumps on the fly

Ivan Villareal — Wed, 17 Feb 2010 20:17:06 +0000

Using pipes and redirection in the shell we can do backups or restore them, performing compression or decompression on the fly, this are the commands I frequently use:

Create a backup with mysqldump and compress the stream with bzip2

ivan@mini:~$ mysqldump –h dbHost –u dbUser –pdbPass –add-drop-database –databases <dbname> | bzip2 > dbName-02-17-2010.sql.bz2dbname>

If I want to restore the backup I use:

ivan@mini:~$ bunzip2 < dbName-02-17-2010.sql.bz2 | mysql –h dbHost –u dbUser –pdbPass

I can switch bzip2 compression by changing only the program name for example gzip/gunzip lzma/unlzma