Re: WIP: a way forward on bootstrap data
John Naylor <jcnaylor@gmail.com>
From: John Naylor <jcnaylor@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2018-03-12T14:25:39Z
Lists: pgsql-hackers
Attachments
- v10-apply-bootstrap-data-patches.sh (application/x-sh)
- v10-0001-Create-infrastructure-for-working-with-the-new-data-.patch (text/x-patch) patch v10-0001
- v10-0002-Hand-edits-of-data-files.patch (text/x-patch) patch v10-0002
- v10-0003-Update-catalog-scripts-to-read-data-files.patch (text/x-patch) patch v10-0003
- v10-0004-Clean-up-header-files-and-update-comments.patch (text/x-patch) patch v10-0004
- v10-0005-Remove-symbols-from-catalog-headers.patch (text/x-patch) patch v10-0005
- v10-0006-Use-default-values-on-more-catalogs.patch (text/x-patch) patch v10-0006
- v10-0007-Implement-OID-macro-lookups.patch (text/x-patch) patch v10-0007
- v10-0008-Generate-pg_type-OID-symbols.patch (text/x-patch) patch v10-0008
- v10-convert_header2dat.pl (application/x-perl)
- v10-rewrite_dat_oid2name.pl (application/x-perl)
- v10-remove_pg_type_oid_symbols.pl (application/x-perl)
I wrote: > I've attached version 9, whose biggest change is to address the above > points of review. I pushed all of the catalog header build logic into > catalog Makefile to avoid creating a separate symbol file. This > involved putting the distprep logic there as well. Enough of the > structure changed that one or two names didn't make sense anymore, so > I changed them. > > As suggested, the conversion script is now part of the patchset and > not committed to the repo. To run the conversion, save everything to a > directory and update the dir vars at the top of > apply-bootstrap-data-patches.sh accordingly. > > A couple things to note that I didn't do: > -With all the new generated headers, the message "Writing ..." is now > quite verbose. It might be worth changing that. > -I'm not sure if I need to change anything involving "make install". > -I haven't tested the MSVC changes. > -I didn't change any clients to actually use the new headers directly. > That might be too ambitious for this cycle anyway. > > While this goes through review, I'll get a head start rebasing the > human readable OIDs and data compaction patches. It didn't take that long to rebase the remaining parts of the patchset, so despite what I said above I went ahead and put them in version 10 (attached), this time via scripted bulk editing rather than as large patches. Changes since the last patchset that contained these parts: -Split out the generation of pg_type OID symbols into its own patch. -Remove single-purpose magic behaviors. -Ditto for the ability to abbreviate attribute names. I decided the added complexity and possible confusion wasn't worth the space savings. -Add some more OID macros for pg_aggregate and pg_range that I missed before. Also, more generally, I cleaned up the apply-patches script and edited its comments and commit messages. Tom Lane wrote: > In 0010, you relabel the types of some OID columns so that genbki.pl > will know which lookup to apply to them. That's not such a problem for > the relabelings that are just macros and genbki.pl converts back to > type OID in the .bki file. But you also did things like s/Oid/regtype/, > and that IS a problem because it will affect what client code sees in > those catalog columns. We've discussed changing those columns to > regfoo types in the past, and decided not to, because of the likelihood > of breaking client queries. I do not think this patch gets to change > that policy. So the way to identify the lookup rule needs to be > independent of whether the column is declared as Oid or an Oid alias type. > Perhaps an explicit marker telling what transformation to make, like > > Oid rngsubtype BKI_LOOKUP(pg_type); > > would work for that. This is also done (now in 0007). -John Naylor
Commits
-
Clarify handling of special-case values in bootstrap catalog data.
- 45c6d75f8cd2 11.0 landed
-
Replace our traditional initial-catalog-data format with a better design.
- 372728b0d495 11.0 landed
-
Faster partition pruning
- 9fdb675fc5d2 11.0 cited
-
Minor cleanup in genbki.pl.
- 8d90b4d01a61 11.0 landed
-
Trivial adjustments in preparation for bootstrap data conversion.
- a351679c806e 11.0 landed
-
Remove hard-coded schema knowledge about pg_attribute from genbki.pl
- 49c784ece766 11.0 landed
-
Minor edits to catalog files and scripts
- 9373baa0f764 11.0 landed
-
Hide most variable-length fields from Form_pg_* structs
- 8137f2c32322 9.2.0 cited