Horton
Menu

Re: increasing the default WAL segment size

David Steele <david@pgmasters.net>

View thread
From: David Steele <david@pgmasters.net>
To: Robert Haas <robertmhaas@gmail.com>, Stephen Frost <sfrost@snowman.net>
Cc: Beena Emerson <memissemerson@gmail.com>, tushar <tushar.ahuja@enterprisedb.com>, Prabhat Sahu <prabhat.sahu@enterprisedb.com>, Ashutosh Sharma <ashu.coek88@gmail.com>, Jim Nasby <Jim.Nasby@bluetreble.com>, Kuntal Ghosh <kuntalghosh.2007@gmail.com>, Andres Freund <andres@anarazel.de>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2017-03-21T22:02:44Z
Lists: pgsql-hackers 
On 3/21/17 3:22 PM, Robert Haas wrote:
> On Tue, Mar 21, 2017 at 9:04 AM, Stephen Frost <sfrost@snowman.net> wrote:
>> In short, I'm also concerned about this change to make WAL file names no
>> longer match up with LSNs and also about the odd stepping that you get
>> as a result of this change when it comes to WAL file names.
>
> OK, that's a bit surprising to me, but what do you want to do about
> it?  If you take the approach that Beena did, then you lose the
> correspondence with LSNs, which is admittedly not great but there are
> already helper functions available to deal with LSN -> filename
> mappings and I assume those will continue to work. If you take the
> opposite approach, then WAL filenames stop being consecutive, which
> seems to me to be far worse in terms of user and tool confusion.

They are already non-consecutive.  Does 000000010000000200000000 really 
logically follow 0000000100000001000000FF?  Yeah, sort of, if you know 
the rules.

> Also
> note that, both currently and with the patch, you can also reduce the
> WAL segment size.  David's proposed naming scheme doesn't handle that
> case, I think, and I think it would be all kinds of a bad idea to use
> one file-naming approach for segments < 16MB and a separate approach
> for segments > 16MB.  That's not making anything easier for users or
> tool authors.

I believe it does handle that case, actually.  The minimum WAL segment 
size is 1MB so they would increase like:

000000010000000100000000
000000010000000100100000
000000010000000100200000
...
0000000100000001FFF00000

You could always calculate the next WAL file by adding 
(wal_seg_size_in_mb << 20) to the previous WAL file's LSN.  This would 
even work for WAL segments > 4GB.  Overall, I think this would make 
calculating WAL ranges simpler than it is now.

The biggest downside I can see is that this would change the naming 
scheme for the default of 16MB compared to previous versions of 
Postgres.  However, for all other wal-seg-size values changes would need 
to be made anyway.

-- 
-David
david@pgmasters.net

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Make WAL segment size configurable at initdb time.

  2. Perform only one ReadControlFile() during startup.

  3. Introduce BYTES unit for GUCs.

  4. Remove useless duplicate inclusions of system header files.

  5. Refactor other replication commands to use DestRemoteSimple.

  6. Add a SHOW command to the replication command language.

  7. Add a new DestReceiver for printing tuples without catalog access.

  8. Support fls().

  9. Extend yesterday's patch making BLCKSZ and RELSEG_SIZE configurable to also

  10. Commit the reasonably uncontroversial parts of J.R. Nield's PITR patch, to

  11. XLOG (also known as WAL -:)) Bootstrap/Startup/Shutdown.

  12. Transaction log manager core code.