Re: directory archive format for pg_dump

On 16.12.2010 17:23, Heikki Linnakangas wrote:
> On 16.12.2010 12:12, Greg Smith wrote:
>> There's a number of small things that I'd like to see improved in new
>> rev of this code
>> ...
>
> In addition to those:
>...

One more thing: the motivation behind this patch is to allow parallel 
pg_dump in the future, so we should be make sure this patch caters well 
for that.

As soon as we have parallel pg_dump, the next big thing is going to be 
parallel dump of the same table using multiple processes. Perhaps we 
should prepare for that in the directory archive format, by allowing the 
data of a single table to be split into multiple files. That way 
parallel pg_dump is simple, you just split the table in chunks of 
roughly the same size, say 10GB each, and launch a process for each 
chunk, writing to a separate file.

It should be a quite simple add-on to the current patch, but will make 
life so much easier for parallel pg_dump. It would also be helpful to 
work around file size limitations on some filesystems.

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com