Horton

Re: WIP: a way forward on bootstrap data

John Naylor <jcnaylor@gmail.com>

View thread View on postgr.es
From: John Naylor <jcnaylor@gmail.com>
To: pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2018-03-27T14:33:07Z
Lists: pgsql-hackers

Attachments

Tom Lane wrote:

>> -Maybe document examples of how to do bulk-editing of data files?
>
> +1.  In the end, that's the reason we're doing all this work, so showing
> people how to benefit seems like a good thing.

I'll hold off on posting a new patchset until I add this to the
documentation, but I wanted to report on a couple of other things:

While adjusting to the 80-column limit, I encountered a separation of
concerns violation between Catalog.pm and reformat_dat_files.pl that I
hadn't noticed before. Fixing that made things easier to read, with
fewer lines of code.

Speaking of bulk editing, that would be done via adopting
reformat_dat_files.pl to the task at hand. I did this myself for two
of the conversion helper scripts. However, enough bitrot has now
occurred that to make the relationship murky. Since I had to adopt
them to the 80-column limit as well, I shaved all the irrelevant
differences away, and now they're just a small diff away from the
reformat script. I also added block comments to help developers find
where they need to edit the script. Since reformat_dat_files.pl has
been substantially altered, I'll attach it here, along with the diffs
to the the helper scripts.

I wrote:

> I’ll see about a new Makefile.

I've attached a draft of this. I thought about adding a call to
duplicate_oids here, but this won't run unless you've run configure
first, and if you've done that, you've likely built already, running
duplicate_oids in the process.

I think I'll consolidate all documentation patches into one, at the
end of the series for maximum flexibility. I liked the idea of
spreading the doc changes over the patches, but there is not a huge
amount of time left.

-John Naylor

Commits

  1. Clarify handling of special-case values in bootstrap catalog data.

  2. Replace our traditional initial-catalog-data format with a better design.

  3. Faster partition pruning

  4. Minor cleanup in genbki.pl.

  5. Trivial adjustments in preparation for bootstrap data conversion.

  6. Remove hard-coded schema knowledge about pg_attribute from genbki.pl

  7. Minor edits to catalog files and scripts

  8. Hide most variable-length fields from Form_pg_* structs