Re: pg_dump directory archive format / parallel pg_dump

On 20.01.2011 17:22, Heikki Linnakangas wrote:
> (I'm working on this, no need to submit a new patch)

Ok, here's a heavily refactored version of this (also available at 
git://git.postgresql.org/git/users/heikki/postgres.git, branch 
pg_dump_directory). The directory format is now identical to the tar 
format, except that in the directory format the files can be compressed. 
Also we don't write the restore.sql file - it would be nice to have, but 
pg_restore doesn't require it. We can leave that as a TODO.

I ended up writing another compression abstraction layer in 
compress_io.c. It wraps fopen / gzopen etc. in a common API, so that the 
caller doesn't need to care if the file is compressed or not. In 
hindsight, the compression API we put in earlier didn't suit us very 
well. But I guess it wasn't a complete waste, as it moved the gory 
details of zlib out of the custom format code.

If compression is used, the files are created with the .gz suffix, and 
include the gzip header so that you can manipulate them easily with 
gzip/gunzip utilities. When reading, we accept files with or without the 
.gz suffix, and you can have some files compressed and others uncompressed.

I haven't updated the documentation yet.

There's one UI thing that bothers me. The option to specify the target 
directory is called --file. But it's clearly not a file. OTOH, I'd hate 
to introduce a parallel --dir option just for this. Any thoughts on this?

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com