Thread

1. How to create directory format backup

   Andrus <kobruleht2@hot.ee> — 2023-02-08T15:00:10Z

   Hi!
   
   Creating backup in directory format using
   
        pg_dump -f "sba" -Fdirectory --jobs=32 --verbose sba
   
   throws error
   
        pg_dump: error: could not stat file "sba/282168.data.gz": value too 
   large
   
   How to fix it ?
   
   Server is Postgres 12 running in Debian Linux 10 under WSL
   Client is pg_dump from Postgres 15 running in Windows 11
   
   Andrus
   

2. Re: How to create directory format backup

   hubert depesz lubaczewski <depesz@depesz.com> — 2023-02-08T17:59:26Z

   On Wed, Feb 08, 2023 at 05:00:10PM +0200, Andrus wrote:
   > Hi!
   > 
   > Creating backup in directory format using
   > 
   >     pg_dump -f "sba" -Fdirectory --jobs=32 --verbose sba
   > 
   > throws error
   > 
   >     pg_dump: error: could not stat file "sba/282168.data.gz": value too
   > large
   > 
   > How to fix it ?
   > 
   > Server is Postgres 12 running in Debian Linux 10 under WSL
   > Client is pg_dump from Postgres 15 running in Windows 11
   
   Looks like your filesystem on client is having limits on file sizes.
   
   Use better filesystem, or just dump on linux, it's filesystems usually
   don't hit these limits.
   
   Best regards,
   
   depesz
   
   
   
   
   

3. Re: How to create directory format backup

   Andrus <kobruleht2@hot.ee> — 2023-02-08T20:59:43Z

   Hi!
   
    >Looks like your filesystem on client is having limits on file sizes. 
   Use better filesystem, or just dump on linux, it's filesystems usually 
   don't hit these limits.
   
   This file size is only 6.2 GB. If custom format is used, pg_dump creates 
   large file without problems. There are no file size limits. Error 
   message is not about this.
   
   Backup client is running in Windows 11 and this cannot changed.
   
   How to create backup in format from which tables can selectively restored?
   
   Posted also in 
   https://stackoverflow.com/questions/75387616/how-to-make-directory-format-backup
   
   Andrus.
   
   

4. Re: How to create directory format backup

   Erik Wienhold <ewie@ewie.name> — 2023-02-08T21:17:58Z

   > On 08/02/2023 21:59 CET Andrus <kobruleht2@hot.ee> wrote:
   >
   > How to create backup in format from which tables can selectively restored?
   
   Dump as custom-format archive (-F custom) and use that with pg_restore and
   options --table or --list/--use-list to select what should be restored.
   
   --
   Erik
   
   
   
   

5. Re: How to create directory format backup

   Andrus <kobruleht2@hot.ee> — 2023-02-08T21:37:26Z

   Hi!
   
   
   >> How to create backup in format from which tables can selectively restored?
   > Dump as custom-format archive (-F custom) and use that with pg_restore and
   > options --table or --list/--use-list to select what should be restored.
   
   How to select tables interactively like pgAdmin allows to select when 
   directory format is used ?
   
   Database contains hundreds of schemas. I need to restore public and 
   other other schema.
   
   Whole backup file is scanned to restore only two schemas. It takes lot 
   of time.
   
   Also directory format allows to use all cores with --jobs=32 parameter. 
   Dump and partial restore using custom format are much slower.
   
   Andrus.
   
   
   > --
   > Erik

6. Re: How to create directory format backup

   Erik Wienhold <ewie@ewie.name> — 2023-02-08T22:45:04Z

   > On 08/02/2023 22:37 CET Andrus <kobruleht2@hot.ee> wrote:
   >
   > > > How to create backup in format from which tables can selectively
   > > > restored?
   > > >
   > > Dump as custom-format archive (-F custom) and use that with pg_restore
   > > and options --table or --list/--use-list to select what should be
   > > restored.
   > >
   > How to select tables interactively like pgAdmin allows to select when
   > directory format is used ?
   >
   > Database contains hundreds of schemas. I need to restore public and other
   > other schema.
   >
   > Whole backup file is scanned to restore only two schemas. It takes lot of
   > time.
   
   pg_dump also accepts options --table and --schema to only dump what you need.
   
   > Also directory format allows to use all cores with --jobs=32 parameter.
   > Dump and partial restore using custom format are much slower.
   
   Run multiple pg_dump processes in parallel where each processes a subset of
   tables with the options mentioned above.
   
   --
   Erik
   
   
   
   

7. Re: How to create directory format backup

   Tom Lane <tgl@sss.pgh.pa.us> — 2023-02-08T22:57:04Z

   Andrus <kobruleht2@hot.ee> writes:
   > This file size is only 6.2 GB. If custom format is used, pg_dump creates 
   > large file without problems. There are no file size limits. Error 
   > message is not about this.
   
   Are you certain that the pg_dump you're using is v15, and not something
   pre-v14?  We got rid of the 4GB limit on stat() results in v14,
   or so we thought.
   
   Anyway, as far as I can find pg_dump can only reach that error message
   via fsync_dir_recurse(), which suggests that a workaround might be
   to add --no-sync so it doesn't try to fsync its output.
   
   			regards, tom lane