Horton

Thread

  1. Configure problem, redux (was Re: TCL installation troubles)

    Tom Lane <tgl@sss.pgh.pa.us> — 1998-10-27T18:25:12Z

    "Thomas G. Lockhart" <lockhart@alumni.caltech.edu> writes:
> I started out with a Makefile.custom which has in it the following:
> USE_TCL= true
> TCL_LIB= -ltcl
> X_LIBS= -L/usr/X11/lib
> TK_LIB= -ltk
> [ and it didn't work ]

There is a comment in Makefile.global.in that says:
	# Please do not edit USE_TCL and USE_TK by hand.
I dunno who put that in or why, but it seems relevant ;-)

An offhand look at configure.in shows that it is also deriving values
for TCL_CONFIG_SH and TK_CONFIG_SH, which need to be propagated into
places that Makefile.custom can't reach.  It looks to me like
mkMakefile.tcldefs.sh is probably not coping gracefully if an empty
string is substituted into it, which is what will happen right now
if USE_TCL isn't set during configure.  By setting USE_TCL in
Makefile.custom, you made the Makefiles think that Tcl support is
set up, but that doesn't take care of the subsidiary files.

> Is someone prepared to fix up the Makefile to make it
> more robust (e.g. it can't include a file which hasn't been constructed
> yet, though there are ways around that by conditionally including it and
> then running make from within that makefile).

Well, *that* is not the problem here.  GNU Make is smarter than you
realize about files that are included into a makefile --- it will
arrange to build the file if missing and then reread the makefile.
The problem is an inconsistent set of configuration values caused
by your hand-setting only some of the needed variables.  And you can't
even fix it by adding the missing variables to Makefile.custom.


This brings up an issue that I had been planning to raise in response
to Brook and Billy's disagreement a couple days ago about how to handle
"libdir" in the Makefiles (thread "Configure problem (and fix)").
Namely: I think we have gotten much too willing to use configure to
rewrite subsidiary files all over the distribution, rather than ensuring
that the configuration decisions are expressed in a central place.

I believe it is considered good Autoconf style to emit a Makefile (or
whatever) that is still hand-editable, so that one can go in and tweak
the configure script's decisions after the fact.  (In other words,
Brook had the right idea as to style: libdir should be left unexpanded
in all references, so that you can actually change it by hand in
Makefile.global if you are so minded.)

We have broken that capability entirely --- if you don't set a parameter
to configure, you have little choice but to go and re-run configure,
because chasing down all of the twenty or so output files that it
generates is not very practical.  (Which was Billy's point, or part of
it; and he's right too.)

This is not good, and it is not necessary.  AFAICS there is *no* good
reason that configure should be rewriting any subdirectory Makefile ---
they all include Makefile.global, so anything they need to know could
be stated, once, in Makefile.global.  With a little more work we could
probably make Makefile.global and config.h be the only files that
configure creates at all.  The various specialized shell scripts that
configure is currently editing could be fixed so that they get their
configuration info from the command line, whence it could be supplied
from Makefile.global via the subdirectory Makefile that is invoking the
script.

I'm not certain how best to handle the SQL scripts that need to know
where libdir is, but I can think of several possibilities, one being
that "create function" could have a library search path built into it,
thus pushing the knowledge of where libdir is into some C code (which
would probably be getting it from the PGLIB environment variable).
Or we could put the value of libdir into a system table somewhere that
the scripts can read at runtime.


It's awfully late to be fixing this stuff for 6.4, unless you want to
slip the release date again.  But I suggest revisiting it for 6.4.1,
and trying to consolidate configuration decisions into as few files
as possible.

			regards, tom lane

  2. Re: Configure problem, redux (was Re: TCL installation troubles)

    Brook Milligan <brook@trillium.nmsu.edu> — 1998-10-27T18:42:35Z

       I'm not certain how best to handle the SQL scripts that need to know
   where libdir is, but I can think of several possibilities, one being
   that "create function" could have a library search path built into it,
   thus pushing the knowledge of where libdir is into some C code (which
   would probably be getting it from the PGLIB environment variable).
   Or we could put the value of libdir into a system table somewhere that
   the scripts can read at runtime.

The way to handle this is to have rules in the Makefile that do the
substitution.  For example, something like the following Makefile
fragment will do the trick, even if the definitition of $libdir in
Makefile.global is modified after configure is run.

     SRCDIR=../../..
     include ${SRCDIR}/Makefile.global
     mklang.sql: mklang.sql.in ${SRCDIR}/Makefile.global
		 sed < $< > $@ -e 's:@libdir@:${libdir}:'

If I remember correctly, this is used in contrib/spi and should be
common practice throughout.

Your suggestion of a single central Makefile.global/config.h to
contain all the configure decisions is a good one.  Together with
rules such as above in the various appropriate places, the problems of
variable expansion and post-configure modifications disappear.
Perhaps not for 6.4, though.

Cheers,
Brook

  3. Re: Configure problem, redux (was Re: TCL installation troubles)

    Thomas Lockhart <lockhart@alumni.caltech.edu> — 1998-10-27T19:06:30Z

    All good points. I had the USE_TCL in my Makefile.custom because for the
last few months/years that was the only way to get anything about tcl
touched by the installation afaik. I'll take it out.

But I'm pretty sure that doesn't explain all the breakage. Will continue
testing a bit (I *really* need to get back to the docs!), but the first
problem I saw was due to a missing file which was not built
automatically, and if you don't do a clean install you won't see the
problem again. That's why I could work with "cvs update -Pd" for weeks
and not see the breakage introduced, because by that point a
Makefile.tcldefs already existed.

> It's awfully late to be fixing this stuff for 6.4, unless you want to
> slip the release date again.  But I suggest revisiting it for 6.4.1,
> and trying to consolidate configuration decisions into as few files
> as possible.

I would strongly suggest that we fix it now, with small incremental
changes to make it work as currently designed. Releasing it broken
doesn't do much good.

So, I'm not sure I understand what the current design is really supposed
to do, but istm that we could do a conditional include of
Makefile.tcldefs, and have Makefile.tcldefs be a prerequisite for
$(PGMS). Like this:

  ifneq ($(wildcard Makefile.tclsh), )
    include Makefile.tclsh
  endif
  ...
  all: Makefile.tclsh
       $(MAKE) $(PGMS)
  ...

Any redesign after v6.4 is released should probably wait for v6.5, since
a v6.4.1 release won't get adequate testing. We managed to break the
Linux port of Postgres for v6.3.1 for similar reasons.

                    - Tom

  4. Re: [HACKERS] Configure problem, redux (was Re: TCL installation troubles)

    Bruce Momjian <maillist@candle.pha.pa.us> — 1998-10-27T20:37:14Z

    > "Thomas G. Lockhart" <lockhart@alumni.caltech.edu> writes:
> > I started out with a Makefile.custom which has in it the following:
> > USE_TCL= true
> > TCL_LIB= -ltcl
> > X_LIBS= -L/usr/X11/lib
> > TK_LIB= -ltk
> > [ and it didn't work ]
> 
> There is a comment in Makefile.global.in that says:
> 	# Please do not edit USE_TCL and USE_TK by hand.
> I dunno who put that in or why, but it seems relevant ;-)
> 
> An offhand look at configure.in shows that it is also deriving values
> for TCL_CONFIG_SH and TK_CONFIG_SH, which need to be propagated into
> places that Makefile.custom can't reach.  It looks to me like
> mkMakefile.tcldefs.sh is probably not coping gracefully if an empty
> string is substituted into it, which is what will happen right now
> if USE_TCL isn't set during configure.  By setting USE_TCL in
> Makefile.custom, you made the Makefiles think that Tcl support is
> set up, but that doesn't take care of the subsidiary files.

Our tcl/tk suff was terrible in earlier releases.  The new code from
Billy is quite nice.  It looks at things the way tcl does.  It looks for
tclConfig.sh in all the normal places, and gets the values from there.

We used to set those values manually, and had all sorts of tcl
version-specific tests that were certain to fail on many platforms.


> > Is someone prepared to fix up the Makefile to make it
> > more robust (e.g. it can't include a file which hasn't been constructed
> > yet, though there are ways around that by conditionally including it and
> > then running make from within that makefile).
> 
> Well, *that* is not the problem here.  GNU Make is smarter than you
> realize about files that are included into a makefile --- it will
> arrange to build the file if missing and then reread the makefile.
> The problem is an inconsistent set of configuration values caused
> by your hand-setting only some of the needed variables.  And you can't
> even fix it by adding the missing variables to Makefile.custom.
> 
> 
> This brings up an issue that I had been planning to raise in response
> to Brook and Billy's disagreement a couple days ago about how to handle
> "libdir" in the Makefiles (thread "Configure problem (and fix)").
> Namely: I think we have gotten much too willing to use configure to
> rewrite subsidiary files all over the distribution, rather than ensuring
> that the configuration decisions are expressed in a central place.
> 
> I believe it is considered good Autoconf style to emit a Makefile (or
> whatever) that is still hand-editable, so that one can go in and tweak
> the configure script's decisions after the fact.  (In other words,
> Brook had the right idea as to style: libdir should be left unexpanded
> in all references, so that you can actually change it by hand in
> Makefile.global if you are so minded.)
> 
> We have broken that capability entirely --- if you don't set a parameter
> to configure, you have little choice but to go and re-run configure,
> because chasing down all of the twenty or so output files that it
> generates is not very practical.  (Which was Billy's point, or part of
> it; and he's right too.)
> 
> This is not good, and it is not necessary.  AFAICS there is *no* good
> reason that configure should be rewriting any subdirectory Makefile ---
> they all include Makefile.global, so anything they need to know could
> be stated, once, in Makefile.global.  With a little more work we could
> probably make Makefile.global and config.h be the only files that
> configure creates at all.  The various specialized shell scripts that
> configure is currently editing could be fixed so that they get their
> configuration info from the command line, whence it could be supplied
> from Makefile.global via the subdirectory Makefile that is invoking the
> script.

I disagree here.  If people want to twiddle, they can after configure. 
People would much rather spell params to configure, rather than to edit
Makefiles.

The use of configure as a centralized test mechanism is a great leap
forward for us.  The basic problem is that Makefile's just can't do the
tests configure can, and any Makefile that needs a test needs a
Makefile.in, and is generated from configure.

Look in Makefile.global.  It is huge, and very hard to understand, even
for veteran developers.  I agree people should be able to twiddle with
Makefile.custom, but they MUST be required to supply the proper flags to
configure.  We can't go down the road of allowing them to avoid the
'configure' flags, and somehow enable things in Makefile.custom.  There
is no logical way to do that, and unless we want to prevent 'configure'
from doing the things it does so well, we will have to live with that
limitation.

I have my configure command and flags in a script and use run it as part
of the cvs update I do to keep my sources current.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

  5. Re: [HACKERS] Re: Configure problem, redux (was Re: TCL installation troubles)

    Bruce Momjian <maillist@candle.pha.pa.us> — 1998-10-27T20:40:54Z

    >    I'm not certain how best to handle the SQL scripts that need to know
>    where libdir is, but I can think of several possibilities, one being
>    that "create function" could have a library search path built into it,
>    thus pushing the knowledge of where libdir is into some C code (which
>    would probably be getting it from the PGLIB environment variable).
>    Or we could put the value of libdir into a system table somewhere that
>    the scripts can read at runtime.
> 
> The way to handle this is to have rules in the Makefile that do the
> substitution.  For example, something like the following Makefile
> fragment will do the trick, even if the definitition of $libdir in
> Makefile.global is modified after configure is run.
> 
>      SRCDIR=../../..
>      include ${SRCDIR}/Makefile.global
>      mklang.sql: mklang.sql.in ${SRCDIR}/Makefile.global
> 		 sed < $< > $@ -e 's:@libdir@:${libdir}:'

The problem here is that you are duplicating the normal configure
processing in every Makefile that needs it.  This will get old very
fast, and hard to maintain.  configure does this already, and
automatically, in one place.  Yes, you must re-run configure, and you do
loose your changes, but pulling all the stuff into every Makefile seems
worse.




-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

  6. Re: [HACKERS] Re: Configure problem, redux (was Re: TCL installation troubles)

    Brook Milligan <brook@trillium.nmsu.edu> — 1998-10-27T21:27:25Z

       The problem here is that you are duplicating the normal configure
   processing in every Makefile that needs it.  This will get old very
   fast, and hard to maintain.  configure does this already, and
   automatically, in one place.  Yes, you must re-run configure, and you do
   loose your changes, but pulling all the stuff into every Makefile seems
   worse.

We are only doing this for the *.sql files because they don't have
within them some ability to do argument expansion.  This is definitely
not good for all the Makefiles or source code, both of which can get
what they need in a single place (Makefile.global or config.h).  I am
not advocating changing that.  In fact, I am agreeing with the idea of
becoming more centralized for those parts of the code that can handle
it, i.e., Makefiles and C source.

For *.sql files there are two choices:

- have configure do the expansion.  This requires either special
  variables or expansion of lots of variables, neither of which is a
  good solution because they require unusual coding practices or
  prevent easy maintenance (i.e., change of Makefile.global) later.

- have make do the expansion.  Certainly this puts some of configure's
  job into some Makefiles, but it offers the advantage that
  Makefile.global can be changed and those changes will propagate
  where they are needed.

It seems like we need a well-documented policy decision, perhaps
between the two following options (plus any others people want to
formulate).

1. we are only speaking (to my knowledge) of the library locations, so
   use ${expanded_libdir} in *.sql files; configure will do the right
   thing with these (i.e., only expand this but leave the "normal"
   variables unexpanded so that changes to Makefile.global propagate
   easily), but changes to Makefile.global cannot touch the configured
   choice for ${expanded_libdir}.  Changes to that require going
   through all the *.sql files seeking those with the wrong
   expansions (or reconfiguring).

2. add rules to those few Makefiles that require it so that make will
   expand for *.sql based on the configured choices.  Changes to
   Makefile.global will do the right thing.

The latter perhaps duplicates a tiny bit of configure, but is more
flexible.  If the rule is made generic (which it really is) it can
even be put into Makefile.global as a pattern target and not even
worried about in any of the Makefiles; if there is a *.sql.in it will
be made into *.sql with the correct substitution, end of story, no
need to be concerned with duplicating parts of configure all over the
place.  This follows the recent trend for a centralized set of rules
for shared library handling.  Why not a central place for the common
*.sql substitution as well, so that what is in Makefile.global gets
propagated?

Cheers,
Brook

  7. Re: [HACKERS] Re: Configure problem, redux (was Re: TCL installation troubles)

    Bruce Momjian <maillist@candle.pha.pa.us> — 1998-10-27T21:33:39Z

    >    The problem here is that you are duplicating the normal configure
>    processing in every Makefile that needs it.  This will get old very
>    fast, and hard to maintain.  configure does this already, and
>    automatically, in one place.  Yes, you must re-run configure, and you do
>    loose your changes, but pulling all the stuff into every Makefile seems
>    worse.
> 
> We are only doing this for the *.sql files because they don't have
> within them some ability to do argument expansion.  This is definitely
> not good for all the Makefiles or source code, both of which can get
> what they need in a single place (Makefile.global or config.h).  I am
> not advocating changing that.  In fact, I am agreeing with the idea of
> becoming more centralized for those parts of the code that can handle
> it, i.e., Makefiles and C source.

Oh.  :-)

I will go back to looking the other way.


-- 
  Bruce Momjian                        |  http://www.op.net/~candle
  maillist@candle.pha.pa.us            |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

  8. Re: [HACKERS] Configure problem, redux (was Re: TCL installation troubles)

    Marc G. Fournier <scrappy@hub.org> — 1998-10-28T01:57:22Z

    On Tue, 27 Oct 1998, Bruce Momjian wrote:

> Look in Makefile.global.  It is huge, and very hard to understand, even
> for veteran developers.  I agree people should be able to twiddle with
> Makefile.custom, but they MUST be required to supply the proper flags to
> configure.  We can't go down the road of allowing them to avoid the
> 'configure' flags, and somehow enable things in Makefile.custom.  There
> is no logical way to do that, and unless we want to prevent 'configure'
> from doing the things it does so well, we will have to live with that
> limitation.
> 
> I have my configure command and flags in a script and use run it as part
> of the cvs update I do to keep my sources current.

	Must agree here...that was what screwed me on the --with-perl :)
I had that set to, and tried to do a 'make install' as myself, which
*should* have worked, except that it tried to install perl stuff too...

	I've never used Makefile.custom myself, as much because I never
remember it is there as that its never been required (configure *should*
handle all that)...

	In fact...other then those that are "veterans", how many ppl out
there even know it exists?  I think its more a legacy feature then
anything...

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org

  9. Re: [HACKERS] Re: Configure problem, redux (was Re: TCL installation troubles)

    Marc G. Fournier <scrappy@hub.org> — 1998-10-28T02:10:06Z

    Just curious, but exactly *what* problem are we trying to fix here?  It
sounds to me like alot of discussion is going into a solution to a problem
that I don't know if exists or not, since I don't recall anyone stating a
problem...

If configure is making 'wrong guesses', then we should be able to tighten
up the tests so that they are correct...

Frankly, and, again, I'm basing this off of memory, I can't think of
anywhere in the 'configure hierarchy' where configure changes anything
major.  Most of the 'changes' are to Makefile.global and config.h (and,
hey, I may not be the one that put the configure seed in, but am I the one
that cursed over a good portion of the tests and extending it)...

So, can someone state *what* the problem is?

On Tue, 27 Oct 1998, Bruce Momjian wrote:

> >    The problem here is that you are duplicating the normal configure
> >    processing in every Makefile that needs it.  This will get old very
> >    fast, and hard to maintain.  configure does this already, and
> >    automatically, in one place.  Yes, you must re-run configure, and you do
> >    loose your changes, but pulling all the stuff into every Makefile seems
> >    worse.
> > 
> > We are only doing this for the *.sql files because they don't have
> > within them some ability to do argument expansion.  This is definitely
> > not good for all the Makefiles or source code, both of which can get
> > what they need in a single place (Makefile.global or config.h).  I am
> > not advocating changing that.  In fact, I am agreeing with the idea of
> > becoming more centralized for those parts of the code that can handle
> > it, i.e., Makefiles and C source.
> 
> Oh.  :-)
> 
> I will go back to looking the other way.
> 
> 
> -- 
>   Bruce Momjian                        |  http://www.op.net/~candle
>   maillist@candle.pha.pa.us            |  (610) 853-3000
>   +  If your life is a hard drive,     |  830 Blythe Avenue
>   +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026
> 

Marc G. Fournier                                
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org

  10. RE: [HACKERS] Re: Configure problem, redux (was Re: TCL installationtroubles)

    Taral <taral@mail.utexas.edu> — 1998-10-28T02:22:06Z

    > So, can someone state *what* the problem is?

AFAICT, there's not a real problem here... Someone was just mentioning that
it isn't easy to override the configure defaults... they appear all OVER the
place. (every *.in file uses them)

The original problem was that the CPPSTDIN code I wrote for configure.in
doesn't detect the case where neither $(CPP) nor $(CPP) - work for cpp from
stdin... we have a possible fix (try /bin/cpp if both fail) which might go
in...

However, I still think that anyone who wants to override configure can
either modify config.cache or config.status... probably the latter in most
cases.

Taral