Horton

Re: WIP: a way forward on bootstrap data

John Naylor <jcnaylor@gmail.com>

View thread View on postgr.es
From: John Naylor <jcnaylor@gmail.com>
To: pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2018-01-30T10:13:25Z
Lists: pgsql-hackers

Attachments

On 1/14/18, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I've got zero faith that the .h files will hold still long enough
> for these patches to be reviewed and applied.  The ones that touch
> significant amounts of data need to be explained as "run this script
> on the current data", rather than presented as static diffs.

For version 7, I've attached a bash script along with the patches
(against master a044378ce2f) that does exactly this. To run, one would
save the patches somewhere and change the PATCH_DIR and REPO_DIR
variables to match.

-I've added MSVC changes, but they are untested.
-I've moved a cosmetic patch up in the series to reduce rebasing
effort. There are some additional comment and style changes.

Not done in this version:

-For the time being I've left out human-readable OIDs and data file
compaction strategies. This is just to reduce effort in rebasing. I'll
add some form of those back after the basics have had serious review.
-Change who is responsible for fmgroids.h. It's debatable whether that
would be a gain anyway.

The README might need to be fleshed out further, possibly with a
separate README for working with the new data format.

>  So, for each catalog header pg_foo.h, there would be a
> generated file, say pg_foo_d.h, containing:
>
> * Natts_ and Anum_ macros for pg_foo
>
> * Any EXPOSE_TO_CLIENT_CODE sections copied from pg_foo.h
>
> * Any OID-value macros for entries in that catalog

This is done (patch 0006). As I mentioned earlier, the sticking point
is the makefiles. I have a working build, but it's not up to project
standards. In particular, for the first attempt I've resorted to
discarding conventions for parallel make safety, so if anyone can
review and offer improvements, I'd be grateful.

> The .dat files need to have header comments that follow project
> conventions, in particular they need to contain copyright statements.
> Likewise for generated files.

Done.

I'll also go ahead and move this to next commitfest.

-John Naylor

Commits

  1. Clarify handling of special-case values in bootstrap catalog data.

  2. Replace our traditional initial-catalog-data format with a better design.

  3. Faster partition pruning

  4. Minor cleanup in genbki.pl.

  5. Trivial adjustments in preparation for bootstrap data conversion.

  6. Remove hard-coded schema knowledge about pg_attribute from genbki.pl

  7. Minor edits to catalog files and scripts

  8. Hide most variable-length fields from Form_pg_* structs