Ivan Villareal » Script to copy data from MySQL to postgreSQL – Ivan Villareal

Script to copy data from MySQL to postgreSQL

Ivan Villareal — Sat, 02 Apr 2011 12:28:32 +0000

This is a perl script I’ve used to move data from mysql 5.1 to postgres 9, this script doesn’t create a schema, it only grabs the data from mysql and try to put it on the same table in postgres.

The only requirement is DBI for mysql and postgres.

 
#!/usr/bin/perl
#
# this script moves all the data from mysql to postgres
# modules required are:
#
# http://search.cpan.org/~timb/DBI-1.616/DBI.pm
#
# please change the dsn, srcUser/srcPass and dstUser/Pass variables.
 
use DBI;
use strict;
 
my $srcUser   = 'ivan';
my $srcPass   = '123';
my $srcDbName = 'datatocopy';
my $srcDbHost = 'mini';
 
my $dstUser  = 'ivan';
my $dstPass  = '456';
my $dstDbName = 'migrateddata';
my $dstDbHost = '192.168.1.145';
 
### do not modify ##
 
my $srcDb  = DBI->connect("dbi:mysql:database=$srcDbName;host=$srcDbHost", $srcUser, $srcPass) or die;
my $dstDb  = DBI->connect("dbi:Pg:database=$dstDbName;host=$dstDbHost", $dstUser, $dstPass) or die;
 
####main########
print "Starting to move data from $srcDbHost to $dstDbHost\n";
$srcDb->do("SET NAMES \'UTF8\'"); 
$dstDb->do("SET CLIENT_ENCODING=\'UTF8\'") or die;
 
 
my $srcTables = $srcDb->selectall_arrayref("SHOW TABLES");
my $dstTables = $dstDb->selectall_hashref("SELECT * FROM pg_tables WHERE NOT tablename ~\'^(pg_|sql_)\'","tablename");
 
my $seqlist = $dstDb->selectall_arrayref("SELECT RELNAME FROM pg_class WHERE relkind=\'S\'");
my $seqmap;
foreach my $s(@$seqlist)  {
    my $seqname = $s->[0];
    if ($seqname=~/^(.*)_([a-z]+)_([a-z]+$)/) {   
        my $item;
        $item->{seq}=$seqname;
        $item->{field}=$2;
        my $table = $1; 
        if (exists $dstTables->{$table}) {   
            push @{$seqmap->{$table}},$item;
        } else {   
            print STDERR "WARN: cannot find table for sequence $seqname\n";
        }   
    }   
}
$dstDb->{AutoCommit}=0;
my $totalRowCount = 0;
foreach my $t(@$srcTables) {  
    my $table = $t->[0];
    unless (exists $dstTables->{$table}) {
        print STDERR "WARN: table $table does not exists in dest db Pg:$dstDbName\n";
        next;
      }
	print "Clearing data from table $table in $dstDbHost\n";
    clear_table($dstDb,$table);
    next if $table=~/^cache/;
    next if $table=~/^locales/;
	print "Copying data from mysql table $table in $srcDbHost to postgres table $table in $dstDbHost\n";
    my $cnt = copy_table($srcDb,$dstDb,$table) unless $table=~/^cache/;
	$totalRowCount = $totalRowCount + $cnt;
    print "$cnt rows copied ($table)\n";
    init_seq($dstDb,$table,$seqmap->{$table}); # if (exists $seqmap->{$table});
  }  
print "Commiting changes \n";     
$dstDb->commit;
print "$totalRowCount rows were copied.\n";
exit 0;
 
 
sub init_seq  {  
    my ($db,$table,$list)=@_;
    return unless $list;
    for my $item (@{$list}) {
        my $ref = $db->selectall_arrayref("SELECT MAX($item->{field}) FROM $table") or die;
        my $val = $ref->[0]->[0];
        if ($val) {
            $db->do("SELECT SETVAL(\'$item->{seq}\',$val)") or die;
          }
      }
}  
 
sub clear_table {  
    my ($dbh,$table)=@_;
    return unless $table;
    $dbh->do("DELETE FROM $table") or die;
}  
 
sub copy_table { 
    my ($srcDb,$dstDb,$table)=@_;
    die unless $table;
    my $slf = $srcDb->prepare("select * from $table limit 1");
    my $rows = $slf->execute() or die;
    return 0 if $rows < 1;
    my $rec1 = $slf->fetchrow_hashref;
    $slf->finish;
    my @fieldnames = sort keys %$rec1;
    my @qm =  map { '?'} @fieldnames;
    my $ins = $dstDb->prepare("INSERT INTO $table (".join(",",@fieldnames).") VALUES(".join(",",@qm).")");
    my $sel = $srcDb->prepare("SELECT * FROM $table");
    $sel->execute or die;
    my $cnt = 0;
    while (my $data = $sel->fetchrow_hashref) {
        $ins->execute(map {$data->{$_}} @fieldnames) or die;
        $cnt++;
      }
    $ins->finish;
    $sel->finish;
    return $cnt;
}

Mysql5 and PHP5 on centos4

Ivan Villareal — Mon, 12 Oct 2009 19:00:06 +0000

I have the need to install php5 and mysql 5 on a centos 4 server, so I found that the Centos Plus repository had this, and this is what I did to get these two working.

First check what is the version of the distro:

cat /etc/*release*

Then make an update:

yum update

After 496 package updates I’ve installed php5

yum --enablerepo=centosplus install php

This were the installed packages:

php-5.1.6-3.el4s1.10
php-cli-5.1.6-3.el4s1.10
php-common-5.1.6-3.el4s1.10

So I’ve created a info.php file, and restarted the apache server:

echo "< ?php phpinfo(); ?>" > /var/www/html/info.php && service httpd restart

It was working correctly, so the next thing was to have mysql 5 in it, so I did:

yum --enablerepo=centosplus install mysql-server

I got some problems here, first I got this:

Transaction Check Error: file /etc/my.cnf from install of mysql-libs-5.0.68-1.el4_6 conflicts with file from package mysql-4.1.22-2.el4
file /usr/share/mysql/charsets/Index.xml from install of mysql-libs-5.0.68-1.el4_6 conflicts with file from package mysql-4.1.22-2.el4
file /usr/share/mysql/charsets/README from install of mysql-libs-5.0.68-1.el4_6 conflicts with file from package mysql-4.1.22-2.el4
file /usr/share/mysql/charsets/armscii8.xml from install of mysql-libs-5.0.68-1.el4_6 conflicts with file from package mysql-4.1.22-2.el4
file /usr/share/mysql/charsets/ascii.xml from install of mysql-libs-5.0.68-1.el4_6 conflicts with file from package mysql-4.1.22-2.el4

So what I found was that yum was trying to install mysql.i386 but the architechture is x86_64, to fix this I had to specify the arch

yum --enablerepo=centosplus install mysql.x86_64 mysql-server

After this I was able to install it, but when I tried to start the server I received some errors about error messages not right, I’ve checked the installed packages and it was installed mysqlclient10, so I removed the error messages problem disappeared, however I still couldn’t startup the server, checking at the logs I found this:

InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
091012 11:01:18  InnoDB: Started; log sequence number 0 0                       091012 11:01:18 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist                                                     091012 11:01:18  mysqld ended

To fix this I cleared the data dir, that had som files, this was in ‘/var/lib/mysql’ and after that I ran mysql_install_db:

[root@prestant /]# cd /var/lib/mysql 
[root@prestant mysql]# rm -rf *
[root@prestant mysql]# mysql_install_db --user=mysql -ldata=/var/lib/mysql
[root@prestant mysql]# service mysqld start

After the server was up, I changed the root password and added a new user:

[root@prestant mysql]# mysqladmin -u root password NEWPASSWORD
[root@prestant mysql]# mysql -pNEWPASSWORD

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.0.82sp1 Source distribution
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
mysql> CREATE USER 'ivan'@'%' IDENTIFIED BY 'passhere';
Query OK, 0 rows affected (0.00 sec)
 
mysql> GRANT ALL PRIVILEGES ON *.* TO 'ivan'@'%' IDENTIFIED BY 'passhere' WITH GRANT OPTION MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0 ;
Query OK, 0 rows affected (0.00 sec)
 
mysql> CREATE USER 'ivan'@'localhost' IDENTIFIED BY 'passhere';
Query OK, 0 rows affected (0.00 sec)
 
mysql> GRANT ALL PRIVILEGES ON *.* TO 'ivan'@'localhost' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)
 
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
 
mysql> exit
Bye

Mysql unique keys and collations

Ivan Villareal — Thu, 08 Oct 2009 01:10:59 +0000

I’m working on an application that stores unique values on a field, so I used the unique index for this column, everything was ok, until I’ve started inserting multi byte characters in it.

The charachters inserted fine, the problem was that I’m using INSERT … ON DUPLICATE KEY UPDATE syntax to avoid looking if the field exists, so after debugging where the application was breaking I got this:

Query failed: Cannot add or update a child row: a foreign key constraint fails 
(`trubaseek/mapKeywordName`, CONSTRAINT `fkDomainMapNameId` FOREIGN KEY (`NameId`) 
REFERENCES `names` (`nameId`) ON DELETE NO ACTION ON UPDATE NO ACTION)<br> 
SQL: INSERT INTO `mapKeywordName` (`keywordId`, `nameId`) VALUES ('1', '326');

so I’ve spent about an hour changing the encodings without success, but after some testing I discovered that hôteles was the same as hotelës, I did several testing from the linux console, the mysql browser, and from a php script, and all revealed the same.

When I did this

INSERT INTO `names` (`name`, `price`, `priceCurrency`, `bids`, `traffic`, `lastUpdate`) 
VALUES ('hotelës', '0', '$US', '0', '0', NOW()) 
ON DUPLICATE KEY UPDATE `price`='0', `priceCurrency`='$US', `bids`='0', `traffic`='0', `lastUpdate` = NOW();

I’ve got this:

Query OK, 2 rows affected (0.01 sec)

Called from a php script mysql_insert_id(); returned me an Id like if it was inserted not updated, but when if I select the returned Id, an error were thrown, because it didn’t existed.

So after some research, I’ve found that adding a collation utf8_bin to that column fixed this issue.

I haven’t tested this enough but for now it appears to be working.