Thread

Loading data into Postgres

Przemyslaw Bak <przemol@st3.makro.com.pl> — 1998-05-31T15:22:37Z

Hi,

every night my script (using perl) will load about 50.000 rows of
data into Postgres. I know two ways:

In first version of my script I used method by inserting, eg:
---------- cut ----------
#!/bin/perl

use Pg;

open (MyData, "...");
while (<MyData>) {
  chop;
  ...
  &insert_cust_sales ($NDate, $CustNo, $ArtGrp, $Qty, $Amount);
}
close MyData;

sub insert_cust_sales {
  my $SQL_QUERY = "insert into cust_sales values ('$_[0]',$_[1],$_[2],$_[3],$_[4],$_[5]);";
  my $conn = Pg::connectdb ("dbname = store3");
  my $result = $conn->exec ( $SQL_QUERY );
}
---------- cut ----------
but it is very slow and my disk works very havy.
So I created another solution, look below:
---------- cut ----------
#!/bin/perl

# Open connection to Postgres
open (PSQL, "| psql store3") || die "Can not open connection to Postgres\n";
print PSQL " copy cust_sales from stdin using delimiters '|';";

# Read data from file
open (MyData, "...");
while (<MyData>) {
  s/	//g; # remove \t
  s/\s+//g;    # ... and spaces
  print PSQL,$_,"\n";
}
print PSQL "\\.\n";
close PSQL;
close MyData;
--------- cut ----------
but while running this script I receive message "resetting connection".
Why ? What's wrong with this script ? I have seen in sources
of postgres that this message is when copy failed, but why ?

przemol