Re: WIP: a way forward on bootstrap data

John Naylor <jcnaylor@gmail.com>

From: John Naylor <jcnaylor@gmail.com>

To: pgsql-hackers <pgsql-hackers@postgresql.org>

Date: 2017-12-13T09:06:35Z

Lists: pgsql-hackers

Attachments

0001_corrections_v1.patch (text/x-patch) patch v1
0002_cleanup_v1.patch (text/x-patch) patch v1
0003_pgattr_schema_isolation_v1.patch (text/x-patch) patch v1
0004A_conversion_scripts_and_headers_v2.patch (text/x-patch) patch v2
0004B_data_files_mechanical_v2.patch.tar.gz (application/x-gzip)
0004C_data_files_hand_edits_v2.patch (text/x-patch) patch v2
0004D_update_distprep_scripts_v2.patch (text/x-patch) patch v2
0005_gut_headers_v2.patch.tar.gz (application/x-gzip)
0006A_human_readable_entries_data_v2.patch.tar.gz (application/x-gzip)
0006B_human_readable_entries_code_v2.patch (text/x-patch) patch v2

There doesn't seem to be any interest in bootstrap data at the moment,
but rather than give up just yet, I've added a couple features to make
a data migration more compelling:

1. Human-readable types, operators, opfamilies, and access methods
2. Column abbreviations

For an example of both practices, an entry from pg_amop changes from

DATA(insert (   1976   21 21 1 s    95  403 0 ));

to

{ opf => 'btree/integer_ops', lt => 'int2', rt => 'int2', str => '1',
oper => '<(int2,int2)', am => 'btree' },


3. Reduce redundancy in pg_proc data by
-computing pronargs from proargtypes and
-leaving out prosrc if it's just a copy of proname.

This, plus a few column abbreviations drastically shrinks pg_proc.dat
line length, even with human-readable types:

pg_proc.h:   avg=175, stdev=25
pg_proc.dat: avg=92,  stdev=43

An example before:

DATA(insert OID = 300 (  float48ne         PGNSP PGUID 12 1 0 0 0 f f
f t t f i s 2 0 16 "700 701" _null_ _null_ _null_ _null_ _null_
float48ne _null_ _null_ _null_ ));

and after:

{ oid => '300',
  n => 'float48ne', lp => 't', p => 's', rt => 'bool', at => 'float4 float8' },

--
I've changed the numbering so that patches with the same number should
be taken as unit, separated only for readability. When referring to
the previous email overview, they map like this:

1-3 : unchanged
4-7 : 4A-4D
8   : N/A - I've left out the SQL generation for now, but I can add it later.

New in this patch set:
Patch 5 rips out the DATA() and DESCR() lines from the headers and
updates the comments to reflect that.
Patches 6A and 6B implement human-readable types etc. as described above.


-John Naylor

Commits

Clarify handling of special-case values in bootstrap catalog data.
- 45c6d75f8cd2 11.0 landed
Replace our traditional initial-catalog-data format with a better design.
- 372728b0d495 11.0 landed
Faster partition pruning
- 9fdb675fc5d2 11.0 cited
Minor cleanup in genbki.pl.
- 8d90b4d01a61 11.0 landed
Trivial adjustments in preparation for bootstrap data conversion.
- a351679c806e 11.0 landed
Remove hard-coded schema knowledge about pg_attribute from genbki.pl
- 49c784ece766 11.0 landed
Minor edits to catalog files and scripts
- 9373baa0f764 11.0 landed
Hide most variable-length fields from Form_pg_* structs
- 8137f2c32322 9.2.0 cited