Thread

1. improved parallel make support

   Peter Eisentraut <peter_e@gmx.net> — 2010-11-02T04:43:27Z

   I have worked on some improvements on how we handle recursive make in
   our makefiles.  Most places uses for loops, which has some
   disadvantages: parallel make doesn't work across directories, make -k
   doesn't work, and make -q doesn't work.  Instead, I went with the
   approach that we already use in the src/backend directory, where we call
   the subordinate makes as target prerequisites.
   
   Note that because with this, parallel make really works, the rule
   dependencies must be correct.  This has always been the case, but now it
   really shows up.  A frequent issue is that this sort of thing no longer
   works:
   
   all: submake-libpgport zic
   
   zic: $(ZICOBJS)
   
   because this relies on the "all" target to execute its prerequisites in
   order.  Instead, you need to write it like this:
   
   all: zic
   
   zic: $(ZICOBJS) | submake-libpgport
   
   (The bar is necessary so that zic isn't considered constantly out of
   date because it depends on a phony target.)
   
   This patch requires GNU make 3.80, because of the above "|" feature and
   the $(eval) function.  Version 3.80 is dated October 2002, so it should
   be no problem, but I do occasionally read of make 3.79 around here;
   maybe it's time to get rid of that.  I did put in a check that makes the
   build fail right away if a wrong version of make is used.
   
   

2. Re: improved parallel make support

   Tom Lane <tgl@sss.pgh.pa.us> — 2010-11-02T14:21:39Z

   Peter Eisentraut <peter_e@gmx.net> writes:
   > This patch requires GNU make 3.80, because of the above "|" feature and
   > the $(eval) function.  Version 3.80 is dated October 2002, so it should
   > be no problem, but I do occasionally read of make 3.79 around here;
   > maybe it's time to get rid of that.  I did put in a check that makes the
   > build fail right away if a wrong version of make is used.
   
   Do we have a handle on how many buildfarm members this will break?
   
   (fwiw, my hpux box is running 3.79.1)
   
   			regards, tom lane
   
   

3. Re: improved parallel make support

   Peter Eisentraut <peter_e@gmx.net> — 2010-11-03T14:34:18Z

   On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote:
   > Peter Eisentraut <peter_e@gmx.net> writes:
   > > This patch requires GNU make 3.80, because of the above "|" feature and
   > > the $(eval) function.  Version 3.80 is dated October 2002, so it should
   > > be no problem, but I do occasionally read of make 3.79 around here;
   > > maybe it's time to get rid of that.  I did put in a check that makes the
   > > build fail right away if a wrong version of make is used.
   > 
   > Do we have a handle on how many buildfarm members this will break?
   
   I suppose we don't.  One way to find out would be to commit just this
   bit
   
   +# We need the $(eval) function, which is available in GNU make 3.80.
   +# That also happens to be the version where the .VARIABLES variable
   +# was introduced, so this is a simple check.
   +ifndef .VARIABLES
   +$(error GNU make 3.80 or newer is required)
   +endif
   
   with a $(warning) instead, and let it run for a bit.
   
   
   
   

4. Re: improved parallel make support

   Tom Lane <tgl@sss.pgh.pa.us> — 2010-11-03T14:42:12Z

   Peter Eisentraut <peter_e@gmx.net> writes:
   > On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote:
   >> Do we have a handle on how many buildfarm members this will break?
   
   > I suppose we don't.  One way to find out would be to commit just this
   > bit
   
   > +# We need the $(eval) function, which is available in GNU make 3.80.
   > +# That also happens to be the version where the .VARIABLES variable
   > +# was introduced, so this is a simple check.
   > +ifndef .VARIABLES
   > +$(error GNU make 3.80 or newer is required)
   > +endif
   
   > with a $(warning) instead, and let it run for a bit.
   
   +1
   
   			regards, tom lane
   
   

5. Re: improved parallel make support

   Peter Eisentraut <peter_e@gmx.net> — 2010-11-06T11:35:30Z

   On ons, 2010-11-03 at 16:34 +0200, Peter Eisentraut wrote:
   > On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote:
   > > Do we have a handle on how many buildfarm members this will break?
   > 
   > I suppose we don't.  One way to find out would be to commit just this
   > bit
   > 
   > +# We need the $(eval) function, which is available in GNU make 3.80.
   > +# That also happens to be the version where the .VARIABLES variable
   > +# was introduced, so this is a simple check.
   > +ifndef .VARIABLES
   > +$(error GNU make 3.80 or newer is required)
   > +endif
   > 
   > with a $(warning) instead, and let it run for a bit.
   
   So far, two machines have reported an older make version:
   
   dawn_bat
   narwhal
   
   both of the mingw type.  Andrew, Dave, could you see about upgrading the
   GNU make installation there?
   
   There are a few machines that haven't build in five days or more, but
   based on their operating system version, it is fairly safe to assume
   that they have an up-to-date version.
   
   
   
   

6. Re: improved parallel make support

   Andrew Dunstan <andrew@dunslane.net> — 2010-11-07T00:39:57Z

   
   On 11/06/2010 07:35 AM, Peter Eisentraut wrote:
   > So far, two machines have reported an older make version:
   >
   > dawn_bat
   > narwhal
   >
   > both of the mingw type.  Andrew, Dave, could you see about upgrading the
   > GNU make installation there?
   
   
   dawn_bat is done.
   
   cheers
   
   andrew
   
   

7. Re: improved parallel make support

   Dave Page <dpage@postgresql.org> — 2010-11-09T11:54:04Z

   On Sat, Nov 6, 2010 at 4:35 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
   > On ons, 2010-11-03 at 16:34 +0200, Peter Eisentraut wrote:
   >> On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote:
   >> > Do we have a handle on how many buildfarm members this will break?
   >>
   >> I suppose we don't.  One way to find out would be to commit just this
   >> bit
   >>
   >> +# We need the $(eval) function, which is available in GNU make 3.80.
   >> +# That also happens to be the version where the .VARIABLES variable
   >> +# was introduced, so this is a simple check.
   >> +ifndef .VARIABLES
   >> +$(error GNU make 3.80 or newer is required)
   >> +endif
   >>
   >> with a $(warning) instead, and let it run for a bit.
   >
   > So far, two machines have reported an older make version:
   >
   > dawn_bat
   > narwhal
   >
   > both of the mingw type.  Andrew, Dave, could you see about upgrading the
   > GNU make installation there?
   
   Narwhal should be OK now.
   
   /D
   
   

8. Re: improved parallel make support

   Peter Eisentraut <peter_e@gmx.net> — 2010-11-10T15:32:48Z

   On tis, 2010-11-09 at 03:54 -0800, Dave Page wrote:
   > Narwhal should be OK now.
   
   The build has issues now, possibly related to the make upgrade.
   
   
   

9. Re: improved parallel make support

   Andrew Dunstan <andrew@dunslane.net> — 2010-11-10T18:13:43Z

   
   On 11/10/2010 10:32 AM, Peter Eisentraut wrote:
   > On tis, 2010-11-09 at 03:54 -0800, Dave Page wrote:
   >> Narwhal should be OK now.
   > The build has issues now, possibly related to the make upgrade.
   >
   >
   
   Yeah, it's complaining about not finding bison, but configure managed to 
   find bison just fine. Are you sure the right make was installed? It 
   looks suspicious because it's not talking about msys virtual maths like 
   the old make did. It needs to be make-3.81-3-msys-1.0.13 
   <http://sourceforge.net/projects/mingw/files/MSYS/make/make-3.81-3/make-3.81-3-msys-1.0.13-bin.tar.lzma/download> 
   You'll need another couple of libraries as well (libiconv and libintl) 
   if they are not already installed. Making this change took me a while to 
   get right on dawn_bat.
   
   cheers
   
   andrew
   
   
   

10. Re: improved parallel make support

    Dave Page <dpage@pgadmin.org> — 2010-11-11T11:58:09Z

    On Wed, Nov 10, 2010 at 6:13 PM, Andrew Dunstan <andrew@dunslane.net> wrote:
    >
    > Yeah, it's complaining about not finding bison, but configure managed to
    > find bison just fine. Are you sure the right make was installed? It looks
    > suspicious because it's not talking about msys virtual maths like the old
    > make did. It needs to be make-3.81-3-msys-1.0.13
    > <http://sourceforge.net/projects/mingw/files/MSYS/make/make-3.81-3/make-3.81-3-msys-1.0.13-bin.tar.lzma/download>
    > You'll need another couple of libraries as well (libiconv and libintl) if
    > they are not already installed. Making this change took me a while to get
    > right on dawn_bat.
    
    I installed the latest make from gnu.org (which I've now uninstalled).
    The Msys installation on this box is old, and doesn't support the lzma
    packages used by the latest releases - and from what I can tell, it
    would take a major upgrade of the installation to get that support.
    I'm not sure thats a path I want to go down, as I have no idea how
    much will break if I do that, and I don't exactly have much in the way
    of spare time to fix it if that happens.
    
    I'm currently leaning towards removing the 9.1 build from the machine;
    on a purely selfish note, I have no interest in mingw/msys builds
    anymore anyway. However, I'm open to suggestions if anyone knows a
    relatively safe way to resolve this.
    
    /D
    
    

11. Re: improved parallel make support

    Andrew Dunstan <andrew@dunslane.net> — 2010-11-11T13:04:08Z

    
    On 11/11/2010 06:58 AM, Dave Page wrote:
    > On Wed, Nov 10, 2010 at 6:13 PM, Andrew Dunstan<andrew@dunslane.net>  wrote:
    >> Yeah, it's complaining about not finding bison, but configure managed to
    >> find bison just fine. Are you sure the right make was installed? It looks
    >> suspicious because it's not talking about msys virtual maths like the old
    >> make did. It needs to be make-3.81-3-msys-1.0.13
    >> <http://sourceforge.net/projects/mingw/files/MSYS/make/make-3.81-3/make-3.81-3-msys-1.0.13-bin.tar.lzma/download>
    >> You'll need another couple of libraries as well (libiconv and libintl) if
    >> they are not already installed. Making this change took me a while to get
    >> right on dawn_bat.
    > I installed the latest make from gnu.org (which I've now uninstalled).
    > The Msys installation on this box is old, and doesn't support the lzma
    > packages used by the latest releases - and from what I can tell, it
    > would take a major upgrade of the installation to get that support.
    > I'm not sure thats a path I want to go down, as I have no idea how
    > much will break if I do that, and I don't exactly have much in the way
    > of spare time to fix it if that happens.
    >
    > I'm currently leaning towards removing the 9.1 build from the machine;
    > on a purely selfish note, I have no interest in mingw/msys builds
    > anymore anyway. However, I'm open to suggestions if anyone knows a
    > relatively safe way to resolve this.
    
    
    No, all you need to unpack those is the basic-bsdtar package. But to 
    save you the pain of all this, I have copied the three objects I 
    installed to get this working on my likewise pretty old Msys to where 
    you can get them. Just grab
    <http://developer.postgresql.org/~adunstan/msys-make.tgz>
    
    As a matter of policy, I do not want to drop support for a FOSS build 
    tool chain on Windows if at all avoidable.
    
    
    cheers
    
    andrew
    
    

12. Re: improved parallel make support

    Dave Page <dpage@pgadmin.org> — 2010-11-11T13:40:55Z

    On Thu, Nov 11, 2010 at 1:04 PM, Andrew Dunstan <andrew@dunslane.net> wrote:
    >
    > No, all you need to unpack those is the basic-bsdtar package.
    
    Ahh, OK. That seems to be in the MinGW (compiler) section of the
    downloads for some reason.
    
    > But to save
    > you the pain of all this, I have copied the three objects I installed to get
    > this working on my likewise pretty old Msys to where you can get them. Just
    > grab
    > <http://developer.postgresql.org/~adunstan/msys-make.tgz>
    
    Thanks - installed.
    
    > As a matter of policy, I do not want to drop support for a FOSS build tool
    > chain on Windows if at all avoidable.
    
    Nor I, however I only have limited time to dedicate to that goal.
    
    
    -- 
    Dave Page
    Blog: http://pgsnake.blogspot.com
    Twitter: @pgsnake
    
    EnterpriseDB UK: http://www.enterprisedb.com
    The Enterprise PostgreSQL Company
    
    

13. Re: improved parallel make support

    Mark Cave-Ayland <mark.cave-ayland@siriusit.co.uk> — 2010-11-11T16:43:22Z

    Dave Page wrote:
    
    > Thanks - installed.
    > 
    >> As a matter of policy, I do not want to drop support for a FOSS build tool
    >> chain on Windows if at all avoidable.
    > 
    > Nor I, however I only have limited time to dedicate to that goal.
    
    One thing to think about is that since PostGIS uses MingW/PGXS on 
    Windows, we use MingW builds in order to generate the Makefiles we need 
    (there is no native MSVC build for Windows). Not being able to do this 
    would cause us great inconvenience :(
    
    
    ATB,
    
    Mark.
    
    -- 
    Mark Cave-Ayland - Senior Technical Architect
    PostgreSQL - PostGIS
    Sirius Corporation plc - control through freedom
    http://www.siriusit.co.uk
    t: +44 870 608 0063
    
    Sirius Labs: http://www.siriusit.co.uk/labs
    
    

14. Re: improved parallel make support

    Andrew Dunstan <andrew@dunslane.net> — 2010-11-11T16:51:22Z

    
    On 11/11/2010 11:43 AM, Mark Cave-Ayland wrote:
    > Dave Page wrote:
    >
    >> Thanks - installed.
    >>
    >>> As a matter of policy, I do not want to drop support for a FOSS 
    >>> build tool
    >>> chain on Windows if at all avoidable.
    >>
    >> Nor I, however I only have limited time to dedicate to that goal.
    >
    > One thing to think about is that since PostGIS uses MingW/PGXS on 
    > Windows, we use MingW builds in order to generate the Makefiles we 
    > need (there is no native MSVC build for Windows). Not being able to do 
    > this would cause us great inconvenience :(
    >
    >
    >
    
    Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and 
    isn't its Windows version build with MSVC?
    
    cheers
    
    andrew
    
    

15. Re: improved parallel make support

    Dave Page <dpage@pgadmin.org> — 2010-11-11T17:03:29Z

    On Thu, Nov 11, 2010 at 4:51 PM, Andrew Dunstan <andrew@dunslane.net> wrote:
    >
    > Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and isn't
    > its Windows version build with MSVC?
    
    Yes - it's a PITA as we have to have a dummy build of the server in
    mingw/msys to compile PostGIS and Slony. We're probably going to be
    looking at that in the not-to-distant future as we want 64bit builds
    of both and will be using VC++.
    
    
    -- 
    Dave Page
    Blog: http://pgsnake.blogspot.com
    Twitter: @pgsnake
    
    EnterpriseDB UK: http://www.enterprisedb.com
    The Enterprise PostgreSQL Company
    
    

16. Re: improved parallel make support

    Mark Cave-Ayland <mark.cave-ayland@siriusit.co.uk> — 2010-11-11T17:19:35Z

    Dave Page wrote:
    
    > On Thu, Nov 11, 2010 at 4:51 PM, Andrew Dunstan <andrew@dunslane.net> wrote:
    >> Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and isn't
    >> its Windows version build with MSVC?
    > 
    > Yes - it's a PITA as we have to have a dummy build of the server in
    > mingw/msys to compile PostGIS and Slony. We're probably going to be
    > looking at that in the not-to-distant future as we want 64bit builds
    > of both and will be using VC++.
    
    Just for the record, a lot of work was done in the 1.4 release series to 
    make MSVC builds possible, and indeed several people have reported success:
    
    http://postgis.refractions.net/pipermail/postgis-devel/2009-March/005102.html
    http://postgis.refractions.net/pipermail/postgis-devel/2010-September/010299.html
    
    The two main outstanding issues as I see it are:
    
    1) The GTK-based GUI for shp2pgsql (although if someone wanted to 
    sponsor work to convert to wxWidgets to bring us in line with pgAdmin, 
    that would be strongly considered).
    
    2) Maintenance of the MSVC build system. So far we have had some 
    complaints about not using MSVC, but then no-one has stepped up to 
    maintain the build system for it. Forcing all existing developers to 
    suddenly start maintaining the Windows build is a total non-starter.
    
    My hope is that one day CMake will enable us to come up with a universal 
    solution, but we're some way from that yet.
    
    
    ATB,
    
    Mark.
    
    -- 
    Mark Cave-Ayland - Senior Technical Architect
    PostgreSQL - PostGIS
    Sirius Corporation plc - control through freedom
    http://www.siriusit.co.uk
    t: +44 870 608 0063
    
    Sirius Labs: http://www.siriusit.co.uk/labs
    
    

17. Re: improved parallel make support

    Dave Page <dpage@pgadmin.org> — 2010-11-11T20:19:47Z

    On Thu, Nov 11, 2010 at 5:19 PM, Mark Cave-Ayland
    <mark.cave-ayland@siriusit.co.uk> wrote:
    > Dave Page wrote:
    >
    >> On Thu, Nov 11, 2010 at 4:51 PM, Andrew Dunstan <andrew@dunslane.net>
    >> wrote:
    >>>
    >>> Interesting. Doesn't EDB's PostgresPlus package include PostGIS, and
    >>> isn't
    >>> its Windows version build with MSVC?
    >>
    >> Yes - it's a PITA as we have to have a dummy build of the server in
    >> mingw/msys to compile PostGIS and Slony. We're probably going to be
    >> looking at that in the not-to-distant future as we want 64bit builds
    >> of both and will be using VC++.
    >
    > Just for the record, a lot of work was done in the 1.4 release series to
    > make MSVC builds possible, and indeed several people have reported success:
    >
    > http://postgis.refractions.net/pipermail/postgis-devel/2009-March/005102.html
    > http://postgis.refractions.net/pipermail/postgis-devel/2010-September/010299.html
    
    Cool - that will help.
    
    > The two main outstanding issues as I see it are:
    >
    > 1) The GTK-based GUI for shp2pgsql (although if someone wanted to sponsor
    > work to convert to wxWidgets to bring us in line with pgAdmin, that would be
    > strongly considered).
    
    :-)
    
    > 2) Maintenance of the MSVC build system. So far we have had some complaints
    > about not using MSVC, but then no-one has stepped up to maintain the build
    > system for it. Forcing all existing developers to suddenly start maintaining
    > the Windows build is a total non-starter.
    
    Unless you're making major architectural changes, it shouldn't take
    any real effort to add/remove the occasional source file. I'm sure
    there are folks that could be persuaded to do that occasionally.
    
    > My hope is that one day CMake will enable us to come up with a universal
    > solution, but we're some way from that yet.
    
    We used CMake for a couple of projects, but ended up abandoning it for
    new stuff. It just didn't work as nicely as we wanted.
    
    -- 
    Dave Page
    Blog: http://pgsnake.blogspot.com
    Twitter: @pgsnake
    
    EnterpriseDB UK: http://www.enterprisedb.com
    The Enterprise PostgreSQL Company
    
    

18. Re: improved parallel make support

    Andrew Dunstan <andrew@dunslane.net> — 2010-11-11T21:12:02Z

    
    On 11/11/2010 03:19 PM, Dave Page wrote:
    >
    >> My hope is that one day CMake will enable us to come up with a universal
    >> solution, but we're some way from that yet.
    > We used CMake for a couple of projects, but ended up abandoning it for
    > new stuff. It just didn't work as nicely as we wanted.
    
    
    Yes, it's been discussed before here too and didn't really go anywhere :-(
    
    cheers
    
    andrew