Horton

Thread

Commits

  1. Fix documentation build with older docbook-xsl

  2. Make documentation builds reproducible

  1. Make documentation builds reproducible

    Peter Eisentraut <peter@eisentraut.org> — 2023-08-23T19:24:07Z

    Somewhere at PGCon, I forgot exactly where, maybe in the same meeting 
where we talked about getting rid of distprep, we talked about that the 
documentation builds are not reproducible (in the sense of 
https://reproducible-builds.org/).  This is easily fixable, the fix is 
available upstream 
(https://github.com/docbook/xslt10-stylesheets/issues/54) but not 
released.  We can backpatch that into our customization layer.  The 
attached patch shows it.

I had actually often wanted this during development.  When making 
documentation tooling changes, it's useful to be able to compare the 
output before and after, and this will eliminate false positives in that.

This patch addresses both the HTML and the FO output.  The man output is 
already reproducible AFAICT.  Note that the final PDF output is 
currently not reproducible; that's a different issue that needs to be 
fixed in FOP.  (See 
https://wiki.debian.org/ReproducibleBuilds/TimestampsInPDFGeneratedByApacheFOP.)

  2. Re: Make documentation builds reproducible

    Tristan Partin <tristan@neon.tech> — 2023-08-24T18:44:34Z

    On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote:
> Somewhere at PGCon, I forgot exactly where, maybe in the same meeting 
> where we talked about getting rid of distprep, we talked about that the 
> documentation builds are not reproducible (in the sense of 
> https://reproducible-builds.org/).  This is easily fixable, the fix is 
> available upstream 
> (https://github.com/docbook/xslt10-stylesheets/issues/54) but not 
> released.  We can backpatch that into our customization layer.  The 
> attached patch shows it.

I am a tiny bit confused here. The commit that solved the issue was 
merged into the master branch in 2018. GitHub lists the lastest release 
as being in 2020. A quick git command shows this has been in releases 
since December of 2018.

	$ git --no-pager tag --contains 0763160
	ndw-test-001
	snapshot-2018-12-07-01
	snapshot-ndw-test/2019-10-04
	snapshot/2018-09-28-172
	snapshot/2018-09-28-173
	snapshot/2018-09-28-174
	snapshot/2018-09-28-175
	snapshot/2018-09-29-176
	snapshot/2018-09-29-177
	snapshot/2018-09-30-178
	snapshot/2018-09-30-179
	snapshot/2018-10-01-180
	snapshot/2018-10-02-183
	snapshot/2018-10-02-184
	snapshot/2018-10-16-185
	snapshot/2018-10-16-186
	snapshot/2018-10-21-188
	snapshot/2018-11-01-191
	snapshot/2019-10-05-bobs
	snapshot/2020-05-28-pdesjardins
	snapshot/2020-06-03

Is there anything I am missing? Is Postgres relying on releases older 
than snapshot-2018-12-07-01? If so, is it possible to up the minimum 
version?

> I had actually often wanted this during development.  When making 
> documentation tooling changes, it's useful to be able to compare the 
> output before and after, and this will eliminate false positives in that.
>
> This patch addresses both the HTML and the FO output.  The man output is 
> already reproducible AFAICT.  Note that the final PDF output is 
> currently not reproducible; that's a different issue that needs to be 
> fixed in FOP.  (See 
> https://wiki.debian.org/ReproducibleBuilds/TimestampsInPDFGeneratedByApacheFOP.)

I think reproducibility is very important. Thanks for taking this on! 

-- 
Tristan Partin
Neon (https://neon.tech)

  3. Re: Make documentation builds reproducible

    Tom Lane <tgl@sss.pgh.pa.us> — 2023-08-24T19:30:06Z

    "Tristan Partin" <tristan@neon.tech> writes:
> On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote:
>> Somewhere at PGCon, I forgot exactly where, maybe in the same meeting 
>> where we talked about getting rid of distprep, we talked about that the 
>> documentation builds are not reproducible (in the sense of 
>> https://reproducible-builds.org/).  This is easily fixable,

> Is there anything I am missing? Is Postgres relying on releases older 
> than snapshot-2018-12-07-01? If so, is it possible to up the minimum 
> version?

AFAICT the "latest stable release" of docbook-xsl is still 1.79.2,
which seems to have been released in 2017, so it's unsurprising that
it's missing this fix.

It's kind of hard to argue that developers (much less distro packagers)
should install unsupported snapshot releases in order to build our docs.
Having said that, maybe we should check whether this patch is compatible
with those snapshot releases, just in case somebody is using one.

			regards, tom lane

  4. Re: Make documentation builds reproducible

    Tristan Partin <tristan@neon.tech> — 2023-08-24T19:52:59Z

    On Thu Aug 24, 2023 at 2:30 PM CDT, Tom Lane wrote:
> "Tristan Partin" <tristan@neon.tech> writes:
> > On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote:
> >> Somewhere at PGCon, I forgot exactly where, maybe in the same meeting 
> >> where we talked about getting rid of distprep, we talked about that the 
> >> documentation builds are not reproducible (in the sense of 
> >> https://reproducible-builds.org/).  This is easily fixable,
>
> > Is there anything I am missing? Is Postgres relying on releases older 
> > than snapshot-2018-12-07-01? If so, is it possible to up the minimum 
> > version?
>
> AFAICT the "latest stable release" of docbook-xsl is still 1.79.2,
> which seems to have been released in 2017, so it's unsurprising that
> it's missing this fix.
>
> It's kind of hard to argue that developers (much less distro packagers)
> should install unsupported snapshot releases in order to build our docs.
> Having said that, maybe we should check whether this patch is compatible
> with those snapshot releases, just in case somebody is using one.

I agree with you. Thanks for the pointer.

-- 
Tristan Partin
Neon (https://neon.tech)

  5. Re: Make documentation builds reproducible

    vignesh C <vignesh21@gmail.com> — 2024-01-20T02:33:01Z

    On Fri, 25 Aug 2023 at 01:23, Tristan Partin <tristan@neon.tech> wrote:
>
> On Thu Aug 24, 2023 at 2:30 PM CDT, Tom Lane wrote:
> > "Tristan Partin" <tristan@neon.tech> writes:
> > > On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote:
> > >> Somewhere at PGCon, I forgot exactly where, maybe in the same meeting
> > >> where we talked about getting rid of distprep, we talked about that the
> > >> documentation builds are not reproducible (in the sense of
> > >> https://reproducible-builds.org/).  This is easily fixable,
> >
> > > Is there anything I am missing? Is Postgres relying on releases older
> > > than snapshot-2018-12-07-01? If so, is it possible to up the minimum
> > > version?
> >
> > AFAICT the "latest stable release" of docbook-xsl is still 1.79.2,
> > which seems to have been released in 2017, so it's unsurprising that
> > it's missing this fix.
> >
> > It's kind of hard to argue that developers (much less distro packagers)
> > should install unsupported snapshot releases in order to build our docs.
> > Having said that, maybe we should check whether this patch is compatible
> > with those snapshot releases, just in case somebody is using one.
>
> I agree with you. Thanks for the pointer.

I'm seeing that there has been no activity in this thread for nearly 5
months, I'm planning to close this in the current commitfest unless
someone is planning to take it forward.

Regards,
Vignesh

  6. Re: Make documentation builds reproducible

    Peter Eisentraut <peter@eisentraut.org> — 2024-01-20T08:32:25Z

    On 20.01.24 03:33, vignesh C wrote:
> On Fri, 25 Aug 2023 at 01:23, Tristan Partin <tristan@neon.tech> wrote:
>>
>> On Thu Aug 24, 2023 at 2:30 PM CDT, Tom Lane wrote:
>>> "Tristan Partin" <tristan@neon.tech> writes:
>>>> On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote:
>>>>> Somewhere at PGCon, I forgot exactly where, maybe in the same meeting
>>>>> where we talked about getting rid of distprep, we talked about that the
>>>>> documentation builds are not reproducible (in the sense of
>>>>> https://reproducible-builds.org/).  This is easily fixable,
>>>
>>>> Is there anything I am missing? Is Postgres relying on releases older
>>>> than snapshot-2018-12-07-01? If so, is it possible to up the minimum
>>>> version?
>>>
>>> AFAICT the "latest stable release" of docbook-xsl is still 1.79.2,
>>> which seems to have been released in 2017, so it's unsurprising that
>>> it's missing this fix.
>>>
>>> It's kind of hard to argue that developers (much less distro packagers)
>>> should install unsupported snapshot releases in order to build our docs.
>>> Having said that, maybe we should check whether this patch is compatible
>>> with those snapshot releases, just in case somebody is using one.
>>
>> I agree with you. Thanks for the pointer.
> 
> I'm seeing that there has been no activity in this thread for nearly 5
> months, I'm planning to close this in the current commitfest unless
> someone is planning to take it forward.

I think there was general agreement with what this patch is doing, but I 
guess it's too boring to actually review the patch in detail.  Let's 
say, if there are no objections, I'll go ahead and commit it.

  7. Re: Make documentation builds reproducible

    Tom Lane <tgl@sss.pgh.pa.us> — 2024-01-20T16:03:07Z

    Peter Eisentraut <peter@eisentraut.org> writes:
> I think there was general agreement with what this patch is doing, but I 
> guess it's too boring to actually review the patch in detail.  Let's 
> say, if there are no objections, I'll go ahead and commit it.

I re-read the thread and have two thoughts:

* We worried about whether this change would be compatible with a
(presently unreleased) version of docbook that contains the upstreamed
fix.  It seems unlikely that there's a problem, but maybe worth
checking?

* I gather that the point here is to change some generated anchor
tags.  Would any of these tags be things people would be likely
to have bookmarked?

			regards, tom lane

  8. Re: Make documentation builds reproducible

    Peter Eisentraut <peter@eisentraut.org> — 2024-01-20T22:44:00Z

    On 20.01.24 17:03, Tom Lane wrote:
> Peter Eisentraut <peter@eisentraut.org> writes:
>> I think there was general agreement with what this patch is doing, but I
>> guess it's too boring to actually review the patch in detail.  Let's
>> say, if there are no objections, I'll go ahead and commit it.
> 
> I re-read the thread and have two thoughts:
> 
> * We worried about whether this change would be compatible with a
> (presently unreleased) version of docbook that contains the upstreamed
> fix.  It seems unlikely that there's a problem, but maybe worth
> checking?

The code in the patch is the same code as upstream, so it would behave 
the same as a new release.

> * I gather that the point here is to change some generated anchor
> tags.  Would any of these tags be things people would be likely
> to have bookmarked?

No, because the problem is that the anchor names are randomly generated 
in each build.

  9. Re: Make documentation builds reproducible

    Tom Lane <tgl@sss.pgh.pa.us> — 2024-01-20T22:59:48Z

    Peter Eisentraut <peter@eisentraut.org> writes:
> On 20.01.24 17:03, Tom Lane wrote:
>> * I gather that the point here is to change some generated anchor
>> tags.  Would any of these tags be things people would be likely
>> to have bookmarked?

> No, because the problem is that the anchor names are randomly generated 
> in each build.

D'oh.  No objection then.

			regards, tom lane

  10. Re: Make documentation builds reproducible

    Peter Eisentraut <peter@eisentraut.org> — 2024-01-22T10:18:31Z

    On 20.01.24 23:59, Tom Lane wrote:
> Peter Eisentraut <peter@eisentraut.org> writes:
>> On 20.01.24 17:03, Tom Lane wrote:
>>> * I gather that the point here is to change some generated anchor
>>> tags.  Would any of these tags be things people would be likely
>>> to have bookmarked?
> 
>> No, because the problem is that the anchor names are randomly generated
>> in each build.
> 
> D'oh.  No objection then.

Thanks, committed.

  11. Re: Make documentation builds reproducible

    Peter Smith <smithpb2250@gmail.com> — 2024-01-23T01:06:34Z

    Hi,

I usually the HTML documentation locally using command:

make STYLE=website html

~

This has been working forever, but seems to have broken due to commit
[1] having an undeclared variable.

e.g.
[postgres@CentOS7-x64 sgml]$ make STYLE=website html
{ \
  echo "<!ENTITY version \"17devel\">"; \
  echo "<!ENTITY majorversion \"17\">"; \
} > version.sgml
'/usr/bin/perl' ./mk_feature_tables.pl YES
../../../src/backend/catalog/sql_feature_packages.txt
../../../src/backend/catalog/sql_features.txt >
features-supported.sgml
'/usr/bin/perl' ./mk_feature_tables.pl NO
../../../src/backend/catalog/sql_feature_packages.txt
../../../src/backend/catalog/sql_features.txt >
features-unsupported.sgml
'/usr/bin/perl' ./generate-errcodes-table.pl
../../../src/backend/utils/errcodes.txt > errcodes-table.sgml
'/usr/bin/perl' ./generate-keywords-table.pl . > keywords-table.sgml
'/usr/bin/perl' ./generate-targets-meson.pl targets-meson.txt
generate-targets-meson.pl > targets-meson.sgml
'/usr/bin/perl'
../../../src/backend/utils/activity/generate-wait_event_types.pl
--docs ../../../src/backend/utils/activity/wait_event_names.txt
/usr/bin/xmllint --nonet --path . --path . --output postgres-full.xml
--noent --valid postgres.sgml
/usr/bin/xsltproc --nonet --path . --path . --stringparam pg.version
'17devel' --param website.stylesheet 1 stylesheet.xsl
postgres-full.xml
runtime error: file stylesheet-html-common.xsl line 452 element if
Variable 'autolink.index.see' has not been declared.
make: *** [html-stamp] Error 10

======
[1] https://github.com/postgres/postgres/commit/b0f0a9432d0b6f53634a96715f2666f6d4ea25a1

Kind Regards,
Peter Smith.
Fujitsu Australia

  12. Re: Make documentation builds reproducible

    Tom Lane <tgl@sss.pgh.pa.us> — 2024-01-23T01:13:00Z

    Peter Smith <smithpb2250@gmail.com> writes:
> I usually the HTML documentation locally using command:
> make STYLE=website html
> This has been working forever, but seems to have broken due to commit
> [1] having an undeclared variable.

Interestingly, that still works fine for me, on RHEL8 with

docbook-dtds-1.0-69.el8.noarch
docbook-style-xsl-1.79.2-9.el8.noarch
docbook-style-dsssl-1.79-25.el8.noarch

What docbook version are you using?

			regards, tom lane

  13. Re: Make documentation builds reproducible

    Peter Smith <smithpb2250@gmail.com> — 2024-01-23T01:32:35Z

    On Tue, Jan 23, 2024 at 12:13 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Peter Smith <smithpb2250@gmail.com> writes:
> > I usually the HTML documentation locally using command:
> > make STYLE=website html
> > This has been working forever, but seems to have broken due to commit
> > [1] having an undeclared variable.
>
> Interestingly, that still works fine for me, on RHEL8 with
>
> docbook-dtds-1.0-69.el8.noarch
> docbook-style-xsl-1.79.2-9.el8.noarch
> docbook-style-dsssl-1.79-25.el8.noarch
>
> What docbook version are you using?
>

[postgres@CentOS7-x64 sgml]$ sudo yum list installed | grep docbook
docbook-dtds.noarch                   1.0-60.el7                 @anaconda
docbook-style-dsssl.noarch            1.79-18.el7                @base
docbook-style-xsl.noarch              1.78.1-3.el7               @anaconda

======
Kind Regards,
Peter Smith.
Fujitsu Australia

  14. Re: Make documentation builds reproducible

    Peter Smith <smithpb2250@gmail.com> — 2024-01-24T22:12:28Z

    On Tue, Jan 23, 2024 at 12:32 PM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Tue, Jan 23, 2024 at 12:13 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >
> > Peter Smith <smithpb2250@gmail.com> writes:
> > > I usually the HTML documentation locally using command:
> > > make STYLE=website html
> > > This has been working forever, but seems to have broken due to commit
> > > [1] having an undeclared variable.
> >
> > Interestingly, that still works fine for me, on RHEL8 with
> >
> > docbook-dtds-1.0-69.el8.noarch
> > docbook-style-xsl-1.79.2-9.el8.noarch
> > docbook-style-dsssl-1.79-25.el8.noarch
> >
> > What docbook version are you using?
> >
>
> [postgres@CentOS7-x64 sgml]$ sudo yum list installed | grep docbook
> docbook-dtds.noarch                   1.0-60.el7                 @anaconda
> docbook-style-dsssl.noarch            1.79-18.el7                @base
> docbook-style-xsl.noarch              1.78.1-3.el7               @anaconda
>

IIUC these releases notes [1] say autolink.index.see existed since
v1.79.1, but unfortunately, that is more recent than my ancient
installed v1.78.1

From the release notes:
------
Robert Stayton: autolink.index.see.xml

New param to control automatic links in index from see and
seealso to indexterm primary.
------

======
[1] https://docbook.sourceforge.net/release/xsl/1.79.1/RELEASE-NOTES.html

Kind Regards,
Peter Smith.
Fujitsu Australia

  15. Re: Make documentation builds reproducible

    Peter Smith <smithpb2250@gmail.com> — 2024-01-29T23:01:25Z

    On Thu, Jan 25, 2024 at 9:12 AM Peter Smith <smithpb2250@gmail.com> wrote:
>
> On Tue, Jan 23, 2024 at 12:32 PM Peter Smith <smithpb2250@gmail.com> wrote:
> >
> > On Tue, Jan 23, 2024 at 12:13 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > >
> > > Peter Smith <smithpb2250@gmail.com> writes:
> > > > I usually the HTML documentation locally using command:
> > > > make STYLE=website html
> > > > This has been working forever, but seems to have broken due to commit
> > > > [1] having an undeclared variable.
> > >
> > > Interestingly, that still works fine for me, on RHEL8 with
> > >
> > > docbook-dtds-1.0-69.el8.noarch
> > > docbook-style-xsl-1.79.2-9.el8.noarch
> > > docbook-style-dsssl-1.79-25.el8.noarch
> > >
> > > What docbook version are you using?
> > >
> >
> > [postgres@CentOS7-x64 sgml]$ sudo yum list installed | grep docbook
> > docbook-dtds.noarch                   1.0-60.el7                 @anaconda
> > docbook-style-dsssl.noarch            1.79-18.el7                @base
> > docbook-style-xsl.noarch              1.78.1-3.el7               @anaconda
> >
>
> IIUC these releases notes [1] say autolink.index.see existed since
> v1.79.1, but unfortunately, that is more recent than my ancient
> installed v1.78.1
>
> From the release notes:
> ------
> Robert Stayton: autolink.index.see.xml
>
> New param to control automatic links in index from see and
> seealso to indexterm primary.
> ------
>
> ======
> [1] https://docbook.sourceforge.net/release/xsl/1.79.1/RELEASE-NOTES.html
>

Is anything going to be changed for this? Since the recent commit [1]
when building the docs now each time I need to first hack (e.g. either
the Makefile or stylesheet-html-common.xml) to declare the missing
‘autolink.index.see’ variable. I know that my old OS is approaching
EOL but I thought my docbook installation was still valid.

======
[1] https://github.com/postgres/postgres/commit/b0f0a9432d0b6f53634a96715f2666f6d4ea25a1

Kind Regards,
Peter Smith.
Fujitsu Australia

  16. Re: Make documentation builds reproducible

    Tom Lane <tgl@sss.pgh.pa.us> — 2024-01-29T23:36:16Z

    Peter Smith <smithpb2250@gmail.com> writes:
>> IIUC these releases notes [1] say autolink.index.see existed since
>> v1.79.1, but unfortunately, that is more recent than my ancient
>> installed v1.78.1

> Is anything going to be changed for this?

I assume Peter E. is going to address it, but FOSDEM is this week and
so a lot of people are going to be busy traveling and conferencing
rather than hacking.  Things might not happen right away.

You could possibly help move things along if you can propose a
workable patch.

			regards, tom lane

  17. Re: Make documentation builds reproducible

    Peter Eisentraut <peter@eisentraut.org> — 2024-02-08T10:47:23Z

    On 23.01.24 02:06, Peter Smith wrote:
> This has been working forever, but seems to have broken due to commit
> [1] having an undeclared variable.

> runtime error: file stylesheet-html-common.xsl line 452 element if
> Variable 'autolink.index.see' has not been declared.
> make: *** [html-stamp] Error 10

I have committed a fix for this.  I have successfully tested docbook-xsl 
1.77.1 through 1.79.*.

  18. Re: Make documentation builds reproducible

    Peter Smith <smithpb2250@gmail.com> — 2024-02-08T23:22:33Z

    On Thu, Feb 8, 2024 at 9:47 PM Peter Eisentraut <peter@eisentraut.org> wrote:
>
> On 23.01.24 02:06, Peter Smith wrote:
> > This has been working forever, but seems to have broken due to commit
> > [1] having an undeclared variable.
>
> > runtime error: file stylesheet-html-common.xsl line 452 element if
> > Variable 'autolink.index.see' has not been declared.
> > make: *** [html-stamp] Error 10
>
> I have committed a fix for this.  I have successfully tested docbook-xsl
> 1.77.1 through 1.79.*.
>

Yes, the latest is working for me now. Thanks!

======
Kind Regards,
Peter Smith.
Fujitsu Australia.