Horton

Re: refactoring basebackup.c

Robert Haas <robertmhaas@gmail.com>

View thread
From: Robert Haas <robertmhaas@gmail.com>
To: Suraj Kharage <suraj.kharage@enterprisedb.com>
Cc: Andres Freund <andres@anarazel.de>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2020-05-13T14:19:41Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Document BaseBackupSync and BaseBackupWrite wait events.

  2. Support long distance matching for zstd compression

  3. Fix possible NULL-pointer-deference in backup_compression.c.

  4. Allow parallel zstd compression when taking a base backup.

  5. Make PostgreSQL::Test::Cluster::run_log() return a useful value.

  6. Fix a few goofs in new backup compression code.

  7. Replace BASE_BACKUP COMPRESSION_LEVEL option with COMPRESSION_DETAIL.

  8. Add 'basebackup_to_shell' contrib module.

  9. Allow extensions to add new backup targets.

  10. Change HAVE_LIBLZ4 and HAVE_LIBZSTD tests to USE_LZ4 and USE_ZSTD.

  11. pg_basebackup: Clean up some bogus file extension tests.

  12. pg_basebackup: Avoid unclean failure with server-compression and -D -.

  13. Fix LZ4 tests for remaining buffer space.

  14. Add support for zstd base backup compression.

  15. pg_basebackup: Allow client-side LZ4 (de)compression.

  16. Add suport for server-side LZ4 base backup compression.

  17. Add min() and max() aggregates for xid8.

  18. Remove superfluous variable.

  19. pg_basebackup: Cleaner handling when compression is multiply specified.

  20. Allow server-side compression to be used with -Fp.

  21. pg_basebackup: Fix a couple of recently-introduced bugs.

  22. Tidy up a few cosmetic issues related to pg_basebackup.

  23. Server-side gzip compression.

  24. Unbreak pg_basebackup/t/010_pg_basebackup.pl on msys

  25. Suppress variable-set-but-not-used warning from clang 13.

  26. Extend the options of pg_basebackup to control compression

  27. Support base backup targets.

  28. Modify pg_basebackup to use a new COPY subprotocol for base backups.

  29. Document that tar archives are now properly terminated.

  30. Fix thinko in bbsink_throttle_manifest_contents.

  31. Have the server properly terminate tar archives.

  32. Minimal fix for unterminated tar archive problem.

  33. Introduce 'bbstreamer' abstraction to modularize pg_basebackup.

  34. Introduce 'bbsink' abstraction to modularize base backup code.

  35. Refactor basebackup.c's _tarWriteDir() function.

  36. Flexible options for CREATE_REPLICATION_SLOT.

  37. Flexible options for BASE_BACKUP.

On Wed, May 13, 2020 at 12:01 AM Suraj Kharage <
suraj.kharage@enterprisedb.com> wrote:

> 8kb 32kb (default value) 128kB 1024kB
> Without refactor patch real 10m22.718s
> user 1m23.629s
> sys 8m51.410s real 8m36.245s
> user 1m8.471s
> sys 7m21.520s real 6m54.299s
> user 0m55.690s
> sys 5m46.502s real 18m3.511s
> user 1m38.197s
> sys 9m36.517s
> With refactor patch (Robert's patch) real 10m11.350s
> user 1m25.038s
> sys 8m39.226s real 8m56.226s
> user 1m9.774s
> sys 7m41.032s real 7m26.678s
> user 0m54.833s
> sys 6m20.057s real 18m17.230s
> user 1m42.749s
> sys 9m53.704s
>
> The above numbers are taken from the minimum of two runs of each scenario.
>
> I can see, when we have TAR_SEND_SIZE as 32kb or 128kb, it is giving us a
> good performance whereas, for 1Mb it is taking 2.5x more time.
>
> Please let me know your thoughts/suggestions on the same.
>

So the patch came out slightly faster at 8kB and slightly slower in the
other tests. That's kinda strange. I wonder if it's just noise. How much do
the results vary run to run?

I would've expected (and I think Andres thought the same) that a smaller
block size would be bad for the patch and a larger block size would be
good, but that's not what these numbers show.

I wouldn't worry too much about the regression at 1MB. Probably what's
happening there is that we're losing some concurrency - perhaps with
smaller block sizes the OS can buffer the entire chunk in the pipe
connecting pg_basebackup to the server and start on the next one, but when
you go up to 1MB it doesn't fit and ends up spending a lot of time waiting
for data to be read from the pipe. Wait event profiling might tell you
more. Probably what this suggests is that you want the largest buffer size
that doesn't cause you to overrun the network/pipe buffer and no larger.
Unfortunately, I have no idea how we'd figure that out dynamically, and I
don't see a reason to believe that everyone will have the same size buffers.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company