Thread

msvc and vista fun

Andrew Dunstan <andrew@dunslane.net> — 2007-06-24T17:23:12Z

I am still very unhappy about the way the MSVC builds work. Although we 
have managed to make it sort of work with the buildfarm script, it is 
distinctly fragile. Last night, for example, I had a build failure due 
to a badly installed zlib. The error state didn't come back to the 
buildfarm script, which carried on merrily to the check stage, which 
also naturally failed, but also didn't manage to give back an error 
state to the buildfarm script, and finally got a failure at the install 
stage. That means that we simply can't rely on this build system to give 
us accurate state on the buildfarm - we might have had a code breakage 
but it won't always tell us that.

Part of the problem, it seems to me, is that this build system is a 
hodge-podge of Perl programs and Windows shell scripts. In at least one 
case we call perl to write a dynamically generated .bat file which we 
then execute. Since the perl is pretty much indispensable (or at least 
would need replacing by an engine of similar capability), I think we 
should make the whole thing a perl suite with some very minimal .bat 
wrappers if necessary.

On a somewhat related note, I have had spectacular lack of success in 
getting either MSVC or MinGW builds to work on Vista - so much so that I 
have currently abandoned my attempts on that platform and I resorted to 
resuscitating an old XP box for testing. Following some advice from 
Magnus, I added ACLs to the build root for both an admin and a non-admin 
user (cacls buildroot /E /T /G AdminUser:C and similarly for a non-admin 
user) . I can build as the admin user but when I come to run initdb it 
fails, complaining that it can't find the postgres executable. If I then 
switch to the non-admin user, it can run initdb just fine. However, that 
user can't build, because it gets a mysterious failure from mt.exe. 
MinGW is even worse - it says it can't run gcc because it can't run 
cc1.exe (IIRC), so it fails at the configure stage! All of this has cost 
me quite a few hours in the last week, with very little to show for it.

Perhaps someone would like to tell me how I can remedy these problems. 
More importantly, this should be in an FAQ or some such. Also, I would 
like to know if we have really tested out on Vista the privilege 
surrendering code that is is supposed to work in Windows. It looks to me 
like it might not be working.

If we can make progress on these issues then I'll be happy. If not, I 
think we need to look carefully at what we say we can support, and what 
we declare to be still experimental.

cheers

andrew

Re: msvc and vista fun

Dave Page <dpage@postgresql.org> — 2007-06-24T18:59:12Z

Andrew Dunstan wrote:

> On a somewhat related note, I have had spectacular lack of success in 
> getting either MSVC or MinGW builds to work on Vista - so much so that I 
> have currently abandoned my attempts on that platform and I resorted to 
> resuscitating an old XP box for testing. Following some advice from 
> Magnus, I added ACLs to the build root for both an admin and a non-admin 
> user (cacls buildroot /E /T /G AdminUser:C and similarly for a non-admin 
> user) . I can build as the admin user but when I come to run initdb it 
> fails, complaining that it can't find the postgres executable. 

Yeah, I ran into that problem as well. I'll look at my Vista box when 
I'm in the office tomorrow and see if I can figure out what hack fixed 
it for me.

If I then
> switch to the non-admin user, it can run initdb just fine. However, that 
> user can't build, because it gets a mysterious failure from mt.exe. 
> MinGW is even worse - it says it can't run gcc because it can't run 
> cc1.exe (IIRC), so it fails at the configure stage! All of this has cost 
> me quite a few hours in the last week, with very little to show for it.

And that one...

> Perhaps someone would like to tell me how I can remedy these problems. 
> More importantly, this should be in an FAQ or some such. Also, I would 
> like to know if we have really tested out on Vista the privilege 
> surrendering code that is is supposed to work in Windows. It looks to me 
> like it might not be working.

What makes you say that?

> If we can make progress on these issues then I'll be happy. If not, I 
> think we need to look carefully at what we say we can support, and what 
> we declare to be still experimental.

Lack of completely reliable buildfarm support for MSVC or trouble 
setting up a buildenv doesn't mean we cannot support a platform - it 
just means we need o be extra vigilent to ensure that regresssion tests 
etc. do actually pass (though that level of failure would be visible on 
the BF I would hope - it certainly has been until now).

Regards, Dave

Re: msvc and vista fun

Magnus Hagander <magnus@hagander.net> — 2007-06-24T19:51:22Z

Andrew Dunstan wrote:
> 
> I am still very unhappy about the way the MSVC builds work. Although we
> have managed to make it sort of work with the buildfarm script, it is
> distinctly fragile. Last night, for example, I had a build failure due
> to a badly installed zlib. The error state didn't come back to the
> buildfarm script, which carried on merrily to the check stage, which
> also naturally failed, but also didn't manage to give back an error

I've heard you report this before, and I've tried and failed many times
to reproduce it. It has *always* come back with the proper errorlevel
when I've tried. So two questions:

1) are you absolutely sure that the problem is not in the buildfarm script?

2) What exactly was the error? So I can try to reproduce that exact
problem here and see if I can find out what it is. What did you do
wrong, and what was the error messages if you still have the log.

> state to the buildfarm script, and finally got a failure at the install
> stage. That means that we simply can't rely on this build system to give
> us accurate state on the buildfarm - we might have had a code breakage
> but it won't always tell us that.

At least it broke eventually :-) But I agree that's a big problem.

> Part of the problem, it seems to me, is that this build system is a
> hodge-podge of Perl programs and Windows shell scripts. In at least one
> case we call perl to write a dynamically generated .bat file which we
> then execute. 

Which one is that? I can't find it. A simple findstr through all the
perl files appears to contain nothing of the sort.

> Since the perl is pretty much indispensable (or at least
> would need replacing by an engine of similar capability), I think we
> should make the whole thing a perl suite with some very minimal .bat
> wrappers if necessary.

That's mostly how it is now, no? What are you referring to specifically?

The only longer ones I can see are:

builddoc.bat - could certainly be put into perl stuff, but it's not
involved in the buildfarm process, so...

clean.bat - doesn't really matter, IMO, and again not involved in the
buildfarm process

vcregress.bat - sure, it could be made perl, but right now you can
actually run the regression tests on an installed system without having
perl at all... But that's not really a requirement, so we could change
that one.

The rest are just thin wrappers.

Speaking of wrappers, are you planning to make the buildfarm use the
modules instead of the perlscripts? IIRC, I made those modules
specifically for the bf - and it should make it even easier to pick up
the errors from there.

I'll leave the Vista parts to Dave since I haven't used it at all there.
These are really completely separate issues, and should be treated as such.

//Magnus

Re: msvc and vista fun

Andrew Dunstan <andrew@dunslane.net> — 2007-06-25T00:24:43Z


Magnus Hagander wrote:
> Andrew Dunstan wrote:
>   
>> I am still very unhappy about the way the MSVC builds work. Although we
>> have managed to make it sort of work with the buildfarm script, it is
>> distinctly fragile. Last night, for example, I had a build failure due
>> to a badly installed zlib. The error state didn't come back to the
>> buildfarm script, which carried on merrily to the check stage, which
>> also naturally failed, but also didn't manage to give back an error
>>     
>
> I've heard you report this before, and I've tried and failed many times
> to reproduce it. It has *always* come back with the proper errorlevel
> when I've tried. So two questions:
>
> 1) are you absolutely sure that the problem is not in the buildfarm script?
>   

Pretty damn sure, yes. This code has functioned correctly on the 
buildfarm on every other platform since its inception.
> 2) What exactly was the error? So I can try to reproduce that exact
> problem here and see if I can find out what it is. What did you do
> wrong, and what was the error messages if you still have the log.
>   

I did indeed keep the logs.

Steps to reproduce: rename your zdll.lib and then run the buildfarm script.

Output:

E:\prog>perl run_build.pl --test
[17:51:47] checking out source ...
[17:52:17] checking if build run needed ...
[17:52:30] copying source to pgsql.3840 ...
3152 File(s) copied
[17:53:44] running configure ...
[17:53:44] running make ...
[18:27:50] running make check ...
[18:27:50] running make install ...
Branch: HEAD
Stage Install failed with status 2


from the bottom of the make log:

Build FAILED.
.\src\backend\utils\adt\oracle_compat.c(898): warning C4090: 'function' 
: different 'const' qualifiers
.\src\backend\utils\adt\oracle_compat.c(900): warning C4090: 'function' 
: different 'const' qualifiers
LINK : fatal error LNK1104: cannot open file 
'e:\prog\pgdepend\zlib\lib\zdll.lib'
LINK : fatal error LNK1104: cannot open file 
'e:\prog\pgdepend\zlib\lib\zdll.lib'
LINK : fatal error LNK1104: cannot open file 
'e:\prog\pgdepend\zlib\lib\zdll.lib'
LINK : fatal error LNK1104: cannot open file 
'e:\prog\pgdepend\zlib\lib\zdll.lib'
LINK : fatal error LNK1104: cannot open file 
'e:\prog\pgdepend\zlib\lib\zdll.lib'
LINK : fatal error LNK1104: cannot open file 
'e:\prog\pgdepend\zlib\lib\zdll.lib'
LINK : fatal error LNK1104: cannot open file 
'e:\prog\pgdepend\zlib\lib\zdll.lib'
LINK : fatal error LNK1104: cannot open file 
'e:\prog\pgdepend\zlib\lib\zdll.lib'
LINK : fatal error LNK1104: cannot open file 
'e:\prog\pgdepend\zlib\lib\zdll.lib'
LINK : fatal error LNK1104: cannot open file 
'e:\prog\pgdepend\zlib\lib\zdll.lib'
LINK : fatal error LNK1104: cannot open file 
'e:\prog\pgdepend\zlib\lib\zdll.lib'
LINK : fatal error LNK1104: cannot open file 
'e:\prog\pgdepend\zlib\lib\zdll.lib'
    2 Warning(s)
    12 Error(s)




Relevant perl code executed by buildfarm:

        chdir "$pgsql/src/tools/msvc";
        @makeout = `build 2>&1`;
        chdir $branch_root;
        my $status = $? >>8;

Could it possibly be that the exit status has only low bits set, and 
isn't what should be returned by wait()? I am checking on this now .... 
(time passes) ... no, the raw status returned is in fact 0.

Now, I see that we do something different in the install case, which is 
why the error gets trapped there. For install, we don't call the .bat 
file at all, we call the install script directly:


        chdir "$pgsql/src/tools/msvc";
        @makeout = `perl install.pl "$installdir" 2>&1`;
        chdir $branch_root;
        my $status = $? >>8;

So it looks like the .bat files are not collecting the exit status and 
passing it on correctly.

>
>   
>> state to the buildfarm script, and finally got a failure at the install
>> stage. That means that we simply can't rely on this build system to give
>> us accurate state on the buildfarm - we might have had a code breakage
>> but it won't always tell us that.
>>     
>
> At least it broke eventually :-) But I agree that's a big problem.
>   

yes.

>
>   
>> Part of the problem, it seems to me, is that this build system is a
>> hodge-podge of Perl programs and Windows shell scripts. In at least one
>> case we call perl to write a dynamically generated .bat file which we
>> then execute. 
>>     
>
> Which one is that? I can't find it. A simple findstr through all the
> perl files appears to contain nothing of the sort.
>   

perl ../../src/tools/msvc/getregress.pl > regress.tmp.bat
call regress.tmp.bat
del regress.tmp.bat


>   
>> Since the perl is pretty much indispensable (or at least
>> would need replacing by an engine of similar capability), I think we
>> should make the whole thing a perl suite with some very minimal .bat
>> wrappers if necessary.
>>     
>
> That's mostly how it is now, no? What are you referring to specifically?
>
> The only longer ones I can see are:
>
> builddoc.bat - could certainly be put into perl stuff, but it's not
> involved in the buildfarm process, so...
>
> clean.bat - doesn't really matter, IMO, and again not involved in the
> buildfarm process
>
> vcregress.bat - sure, it could be made perl, but right now you can
> actually run the regression tests on an installed system without having
> perl at all... But that's not really a requirement, so we could change
> that one.
>   

perl2exe ?

> The rest are just thin wrappers.
>   

Well, my .bat-fu was last in use around DOS6.2, so I'm not mucjh help 
with them :-)  I don't think they're really thin enough. Why not just 
have them call "perl scriptname args"?

>
> Speaking of wrappers, are you planning to make the buildfarm use the
> modules instead of the perlscripts? IIRC, I made those modules
> specifically for the bf - and it should make it even easier to pick up
> the errors from there.
>
>
>
>   

Yes, you did. But as I think I told you, I had a rethink about that. The 
whole point about the buildfarm is to do things as closely to what a 
human would do as possible, so I don't want to make it use the perl 
modules like that unless we really really have to. I also want to have 
as little special casing as possible in the buildfarm.


cheers

andrew

Re: msvc and vista fun

Andrew Dunstan <andrew@dunslane.net> — 2007-06-25T01:22:13Z


Dave Page wrote:
>
>> Perhaps someone would like to tell me how I can remedy these 
>> problems. More importantly, this should be in an FAQ or some such. 
>> Also, I would like to know if we have really tested out on Vista the 
>> privilege surrendering code that is is supposed to work in Windows. 
>> It looks to me like it might not be working.
>
> What makes you say that?


It was just a suspicion, but maybe the fact that the admin user can't 
run while the non-admin user can indicates that it is working (possibly 
too well).

>
>> If we can make progress on these issues then I'll be happy. If not, I 
>> think we need to look carefully at what we say we can support, and 
>> what we declare to be still experimental.
>
> Lack of completely reliable buildfarm support for MSVC or trouble 
> setting up a buildenv doesn't mean we cannot support a platform - it 
> just means we need o be extra vigilent to ensure that regresssion 
> tests etc. do actually pass (though that level of failure would be 
> visible on the BF I would hope - it certainly has been until now).
>
>

I am not only concerned about the buildfarm, but about the robustness, 
maintainability and simplicity of the build process. Having to have a 
completely separate build system for msvc is really quite sucky.

cheers

andrew

Re: msvc and vista fun

Michael Paesold <mpaesold@gmx.at> — 2007-06-25T06:07:26Z

Andrew Dunstan wrote:

> Relevant perl code executed by buildfarm:
> 
>        chdir "$pgsql/src/tools/msvc";
>        @makeout = `build 2>&1`;
>        chdir $branch_root;
>        my $status = $? >>8;

I know the docs say otherwise, but would it be possible that "chdir" 
somehow resets $? on windows, sometimes, under some circumstances?

Perhaps just move up the "my $status.." one line up?

Best Regards
Michael Paesold

Re: msvc and vista fun

Zeugswetter Andreas OSB sIT <andreas.zeugswetter@s-itsolutions.at> — 2007-06-25T07:19:37Z

> user) . I can build as the admin user but when I come to run 
> initdb it fails, complaining that it can't find the postgres 
> executable.

FYI, this happens on my Win 2000 also.
Maybe a problem with mixed / \ path separators after RestrictExec.

Andreas

Re: msvc and vista fun

Dave Page <dpage@postgresql.org> — 2007-06-25T08:14:08Z

Andrew Dunstan wrote:
> 
> 
> Dave Page wrote:
>>
>>> Perhaps someone would like to tell me how I can remedy these 
>>> problems. More importantly, this should be in an FAQ or some such. 
>>> Also, I would like to know if we have really tested out on Vista the 
>>> privilege surrendering code that is is supposed to work in Windows. 
>>> It looks to me like it might not be working.
>>
>> What makes you say that?
> 
> 
> It was just a suspicion, but maybe the fact that the admin user can't 
> run while the non-admin user can indicates that it is working (possibly 
> too well).

Hmm, yes - I see what you mean. How are you defining 'admin user'? When 
I ran into this problem from what I recall I was originally trying to 
swim against the Vista security model by using the Administrator account 
  which is disabled by default (for anyone still doubting that from the 
last discussion on the topic, please see 
http://blogs.msdn.com/windowsvistasecurity/archive/2006/08/27/windowsvistasecurity-.aspx). 
It went away when I started using my own user account which is not 'the' 
administrator, but is a member of the administrators group.

> I am not only concerned about the buildfarm, but about the robustness, 
> maintainability and simplicity of the build process. Having to have a 
> completely separate build system for msvc is really quite sucky.

Well, yes, but unfortunately there's not really any other way to get a 
halfway decent development environment on Windows. Mingw/Msys are 
horrendously maintained - mingw still doesn't work without hacking on 
Vista as you've found, and msys doesn't work at all on 64 bit versions 
for example (at least it didn't the last few times I tried - it's 
possible they've fixed it now). The debugger doesn't work for most 
people, and the development environment is just plain nasty to keep up 
to date.

Now 99.99% of users don't build on Windows anyway, so I'm not so much 
concerned about their ability to maintain and use the build environment 
as ours - if that means we have to fix a few bugs in the MSVC scripts 
during the dev cycle, so be it. The setup I'm using now for 8.3 is 
already orders of magnitude easier to setup and maintain than the 
mingw/msys environment, plus we have a decent debugger (and other tools) 
now, and the ability to run on 64 bit platforms, not to mention a 
compiler with which we can (eventually) build for them as well.

Regards, Dave

Re: msvc and vista fun

Dave Page <dpage@postgresql.org> — 2007-06-25T08:20:59Z

Dave Page wrote:
> If I then
>> switch to the non-admin user, it can run initdb just fine. However, 
>> that user can't build, because it gets a mysterious failure from 
>> mt.exe. MinGW is even worse - it says it can't run gcc because it 
>> can't run cc1.exe (IIRC), so it fails at the configure stage! All of 
>> this has cost me quite a few hours in the last week, with very little 
>> to show for it.
> 
> And that one...

See http://aarongiles.com/?p=199 for more details, but the quick fix I 
used was to add /mingw/libexec/gcc/mingw32/3.4.2 to the path in 
/etc/profile.

Regards, Dave

Re: msvc and vista fun

Magnus Hagander <magnus@hagander.net> — 2007-06-25T08:43:53Z

On Sun, Jun 24, 2007 at 08:24:43PM -0400, Andrew Dunstan wrote:
> 
> 
> Magnus Hagander wrote:
> >Andrew Dunstan wrote:
> >  
> >>I am still very unhappy about the way the MSVC builds work. Although we
> >>have managed to make it sort of work with the buildfarm script, it is
> >>distinctly fragile. Last night, for example, I had a build failure due
> >>to a badly installed zlib. The error state didn't come back to the
> >>buildfarm script, which carried on merrily to the check stage, which
> >>also naturally failed, but also didn't manage to give back an error
> >>    
> >
> >I've heard you report this before, and I've tried and failed many times
> >to reproduce it. It has *always* come back with the proper errorlevel
> >when I've tried. So two questions:
> >
> >1) are you absolutely sure that the problem is not in the buildfarm script?
> >  
> 
> Pretty damn sure, yes. This code has functioned correctly on the 
> buildfarm on every other platform since its inception.

I think you are overestimating the cross-platformness of perl...  :-)

> >2) What exactly was the error? So I can try to reproduce that exact
> >problem here and see if I can find out what it is. What did you do
> >wrong, and what was the error messages if you still have the log.
> >  
> 
> I did indeed keep the logs.
> 
> Steps to reproduce: rename your zdll.lib and then run the buildfarm script.

Ok.

Running this *outside* the buildfarm properly sets errorlevel=1 when it
exits.

Running the same test from the buildfarm script on the same machine, and it
picks up the error and reports it just fine.
(http://pgbuildfarm.org/cgi-bin/show_log.pl?nm=skylark&dt=2007-06-25%2008:28:31)

Since nobody else (AFAIK) has seen this, I'd be inclined to blame something
in your environment. 

Can you run the build in "broken mode" but *without* using the buildfarm
scripts, and see what errorlevel you get? Meaning:
build.bat
... wait ...
echo %errorlevel%

It should output 1 if that part works, 0 if it fails. That'll tell us if
the problem is in the bf or if it's in the actual build system.

> >>Part of the problem, it seems to me, is that this build system is a
> >>hodge-podge of Perl programs and Windows shell scripts. In at least one
> >>case we call perl to write a dynamically generated .bat file which we
> >>then execute. 
> >>    
> >
> >Which one is that? I can't find it. A simple findstr through all the
> >perl files appears to contain nothing of the sort.
> >  
> 
> perl ../../src/tools/msvc/getregress.pl > regress.tmp.bat
> call regress.tmp.bat
> del regress.tmp.bat

Ah. I was looking at the build parts only, not the regress part. My bad.

Anyway, yes, the vcregress.bat file was the only one on the list that made
sense to consider redoing in perl.

> >>Since the perl is pretty much indispensable (or at least
> >>would need replacing by an engine of similar capability), I think we
> >>should make the whole thing a perl suite with some very minimal .bat
> >>wrappers if necessary.
> >>    
> >
> >That's mostly how it is now, no? What are you referring to specifically?
> >
> >The only longer ones I can see are:
> >
> >builddoc.bat - could certainly be put into perl stuff, but it's not
> >involved in the buildfarm process, so...
> >
> >clean.bat - doesn't really matter, IMO, and again not involved in the
> >buildfarm process
> >
> >vcregress.bat - sure, it could be made perl, but right now you can
> >actually run the regression tests on an installed system without having
> >perl at all... But that's not really a requirement, so we could change
> >that one.
> >  
> 
> perl2exe ?

No way, that's just too ugly :-) Since you basically ship perl packaged
inside the exe anyway, we might as well accept the requirement of perl to
run the regression tests. Actually, ´given the above we *already* require
perl for the regression tests, I had forgotten about that part.

> 
> >The rest are just thin wrappers.
> >  
> 
> Well, my .bat-fu was last in use around DOS6.2, so I'm not mucjh help 
> with them :-)  I don't think they're really thin enough. Why not just 
> have them call "perl scriptname args"?

I guess you could. The point of the split in build.bat today is basically
that you run mkvcbuild separately if you want to work from the GUI. But
that could of course be dealt with with commandline parameters if there's
really a benefit in it. But I'm not sure there is - there's very little
logic in the bat files, and that's what matters IMO.

> >Speaking of wrappers, are you planning to make the buildfarm use the
> >modules instead of the perlscripts? IIRC, I made those modules
> >specifically for the bf - and it should make it even easier to pick up
> >the errors from there.
> >
> >
> 
> Yes, you did. But as I think I told you, I had a rethink about that. The 
> whole point about the buildfarm is to do things as closely to what a 
> human would do as possible, so I don't want to make it use the perl 
> modules like that unless we really really have to. I also want to have 
> as little special casing as possible in the buildfarm.

Ah, you did tell me that. Sorry, forgot about that. Fair 'nuff.

//Magnus

Re: msvc and vista fun

Dave Page <dpage@postgresql.org> — 2007-06-25T09:00:16Z

Magnus Hagander wrote:
> Running the same test from the buildfarm script on the same machine, and it
> picks up the error and reports it just fine.
> (http://pgbuildfarm.org/cgi-bin/show_log.pl?nm=skylark&dt=2007-06-25%2008:28:31)

I ran on Baiji (renaming the zlib directory, instead of the .lib as 
Magnus did) and see a error as well: 
http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=baiji&dt=2007-06-25%2008:36:47

Just for kicks, that was also on Vista. I'm going to run on my Win2K3 
animal in a minute - that one runs VC++ Express, unlike Baiji and 
Skylark which I believe are both 'proper' Visual Studio. Will report 
back when it's done.

Regards, Dave

Re: msvc and vista fun

Dave Page <dpage@postgresql.org> — 2007-06-25T10:06:37Z

Dave Page wrote:
> I'm going to run on my Win2K3 
> animal in a minute - that one runs VC++ Express, unlike Baiji and 
> Skylark which I believe are both 'proper' Visual Studio. Will report 
> back when it's done.

Yep, it failed at make as well, and reported it appropriately to the BF.

Regards, Dave

Re: msvc and vista fun

Andrew Dunstan <andrew@dunslane.net> — 2007-06-25T13:13:51Z


Dave Page wrote:
> Dave Page wrote:
>> I'm going to run on my Win2K3 animal in a minute - that one runs VC++ 
>> Express, unlike Baiji and Skylark which I believe are both 'proper' 
>> Visual Studio. Will report back when it's done.
>
> Yep, it failed at make as well, and reported it appropriately to the BF.
>
>

Well, that's *very* curious. I have seen this on more than one machine.

I'll run the test Magnus suggests and see what we can discover.

cheers

andrew

Re: msvc and vista fun

Andrew Dunstan <andrew@dunslane.net> — 2007-06-25T14:06:57Z

Magnus Hagander wrote:
>
> Can you run the build in "broken mode" but *without* using the buildfarm
> scripts, and see what errorlevel you get? Meaning:
> build.bat
> ... wait ...
> echo %errorlevel%
>
> It should output 1 if that part works, 0 if it fails. That'll tell us if
> the problem is in the bf or if it's in the actual build system.
>
>
>   

It fails and says 1.

Now, if I change the build script so it says

  exit %E

instead of

  exit /b %E

I pick up the exit status just fine. I wonder if wer have different 
flavors of command interpreter? (My perl is the latest one from 
ActiveState, btw. I assume you're using that.)

Can we change that or make it switchable? I'd be happy to provide an 
environment variable like RUNNING_BUILDFARM if that would help.

cheers

andrew

Re: msvc and vista fun

Magnus Hagander <magnus@hagander.net> — 2007-06-25T14:14:15Z

On Mon, Jun 25, 2007 at 10:06:57AM -0400, Andrew Dunstan wrote:
> 
> 
> Magnus Hagander wrote:
> >
> >Can you run the build in "broken mode" but *without* using the buildfarm
> >scripts, and see what errorlevel you get? Meaning:
> >build.bat
> >... wait ...
> >echo %errorlevel%
> >
> >It should output 1 if that part works, 0 if it fails. That'll tell us if
> >the problem is in the bf or if it's in the actual build system.
> >
> >
> >  
> 
> It fails and says 1.
> 
> Now, if I change the build script so it says
> 
>  exit %E
> 
> instead of
> 
>  exit /b %E

That's just weird :-( And making that change makes the script unusable from
the commandline.

> I pick up the exit status just fine. I wonder if wer have different 
> flavors of command interpreter? (My perl is the latest one from 
> ActiveState, btw. I assume you're using that.)

cmd.exe from Windows - I assume you haven't installed some funky addon
special command interpreter? In that case, it should be the same. So I
have:
Microsoft Windows [Version 5.2.3790]
(C) Copyright 1985-2003 Microsoft Corp.

As for perl, I'm probably not on the very latest, but it's not so old. I'm
on:
This is perl, v5.8.8 built for MSWin32-x64-multi-thread
(with 33 registered patches, see perl -V for more detail)

Dave - what version are you on?

> Can we change that or make it switchable? I'd be happy to provide an 
> environment variable like RUNNING_BUILDFARM if that would help.

We could, but it seems very ugly. And again, it's *not* required for
buildfarm on my or Daves machines. So I'd rather like to know why it's
actually happening than just blindly change it.

//Magnus

Re: msvc and vista fun

Dave Page <dpage@postgresql.org> — 2007-06-25T14:33:56Z

Magnus Hagander wrote:
> As for perl, I'm probably not on the very latest, but it's not so old. I'm
> on:
> This is perl, v5.8.8 built for MSWin32-x64-multi-thread
> (with 33 registered patches, see perl -V for more detail)
> 
> 
> Dave - what version are you on?

This is perl, v5.8.8 built for MSWin32-x86-multi-thread
(with 50 registered patches, see perl -V for more detail)

/D

Re: msvc and vista fun

Andrew Dunstan <andrew@dunslane.net> — 2007-06-25T14:45:00Z

Magnus Hagander wrote:
> cmd.exe from Windows - I assume you haven't installed some funky addon
> special command interpreter? In that case, it should be the same. So I
> have:
> Microsoft Windows [Version 5.2.3790]
> (C) Copyright 1985-2003 Microsoft Corp.
>
>
>   
>   

Hmm. Mine says 5.1.2600, which seems to be quite old. I'll see about 
updating ... the strange thing is I'm pretty sure I've seen this on 
other more up to date machines.

cheers

andrew

Re: msvc and vista fun

Dave Page <dpage@postgresql.org> — 2007-06-25T14:50:49Z

Andrew Dunstan wrote:

> Hmm. Mine says 5.1.2600, which seems to be quite old. I'll see about 
> updating ... the strange thing is I'm pretty sure I've seen this on 
> other more up to date machines.

You can't update that unless you upgrade to Windows 2003 or Vista. 
That's the XP command shell.

Regards, Dave

Re: msvc and vista fun

Magnus Hagander <magnus@hagander.net> — 2007-06-25T14:50:58Z

On Mon, Jun 25, 2007 at 10:45:00AM -0400, Andrew Dunstan wrote:
> 
> 
> Magnus Hagander wrote:
> >cmd.exe from Windows - I assume you haven't installed some funky addon
> >special command interpreter? In that case, it should be the same. So I
> >have:
> >Microsoft Windows [Version 5.2.3790]
> >(C) Copyright 1985-2003 Microsoft Corp.
> >
> >
> >  
> >  
> 
> Hmm. Mine says 5.1.2600, which seems to be quite old. I'll see about 
> updating ... the strange thing is I'm pretty sure I've seen this on 
> other more up to date machines.

That's because you are on XP. 5.1 is XP, 5.2 is 2003. To make things more
interesting, XP x64 is 5.2...

So it might be worthwhile to see if this is something that happens on <=
XP, and works on >= 2003. Dave, did you test on any server OS other than
2003, or any client that's XP or earlier?

//Magnus

Re: msvc and vista fun

Andrew Dunstan <andrew@dunslane.net> — 2007-06-25T15:06:45Z


Dave Page wrote:
> Dave Page wrote:
>> If I then
>>> switch to the non-admin user, it can run initdb just fine. However, 
>>> that user can't build, because it gets a mysterious failure from 
>>> mt.exe. MinGW is even worse - it says it can't run gcc because it 
>>> can't run cc1.exe (IIRC), so it fails at the configure stage! All of 
>>> this has cost me quite a few hours in the last week, with very 
>>> little to show for it.
>>
>> And that one...
>
> See http://aarongiles.com/?p=199 for more details, but the quick fix I 
> used was to add /mingw/libexec/gcc/mingw32/3.4.2 to the path in 
> /etc/profile.
>
>


That seems to have fixed it. But now of cousre I forgot that the latest 
version is broken w.r.t. gettimeofday ... *sigh*

cheers

andrew

Re: msvc and vista fun

Andrew Dunstan <andrew@dunslane.net> — 2007-06-25T15:25:35Z


Magnus Hagander wrote:
>> Can we change that or make it switchable? I'd be happy to provide an 
>> environment variable like RUNNING_BUILDFARM if that would help.
>>     
>
> We could, but it seems very ugly. And again, it's *not* required for
> buildfarm on my or Daves machines. So I'd rather like to know why it's
> actually happening than just blindly change it.
>
>
>   

Not surprisingly, I am not the only person who has had this problem. See 
http://jira.codehaus.org/browse/CONTINUUM-413 where the suggested 
solution is:

  if "%MAVEN_TERMINATE_CMD%" == "on" exit %ERRORLEVEL%
  exit /b %ERRORLEVEL%

There are 12 lines involved in our .bat files:

build.bat:35:exit /b %E%
builddoc.bat:40:exit /b
builddoc.bat:50:exit /b
install.bat:9:exit /b 1
install.bat:18:exit /b %ERRORLEVEL%
vcregress.bat:35:   if errorlevel 1 exit /b 1
vcregress.bat:51:exit /b %E%
vcregress.bat:64:   if errorlevel 1 exit /b 1
vcregress.bat:66:   if errorlevel 1 exit /b 1
vcregress.bat:86:exit /b %E%
vcregress.bat:97:if %CONTRIBERROR%==1 exit /b 1
vcregress.bat:113:exit /b %E%

Would making a change like this in those 12 places be so ugly?

cheers

andrew

Re: msvc and vista fun

Dave Page <dpage@postgresql.org> — 2007-06-25T15:38:57Z

Magnus Hagander wrote:
> So it might be worthwhile to see if this is something that happens on <=
> XP, and works on >= 2003. Dave, did you test on any server OS other than
> 2003, or any client that's XP or earlier?

No, my testing was only on 2k3 and vista.

/D

Re: msvc and vista fun

Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk> — 2007-06-25T19:18:17Z

On Sun, 2007-06-24 at 13:23 -0400, Andrew Dunstan wrote:

(cut)

> On a somewhat related note, I have had spectacular lack of success in 
> getting either MSVC or MinGW builds to work on Vista - so much so that I 
> have currently abandoned my attempts on that platform and I resorted to 
> resuscitating an old XP box for testing. Following some advice from 
> Magnus, I added ACLs to the build root for both an admin and a non-admin 
> user (cacls buildroot /E /T /G AdminUser:C and similarly for a non-admin 
> user) . I can build as the admin user but when I come to run initdb it 
> fails, complaining that it can't find the postgres executable. If I then 
> switch to the non-admin user, it can run initdb just fine. However, that 
> user can't build, because it gets a mysterious failure from mt.exe. 
> MinGW is even worse - it says it can't run gcc because it can't run 
> cc1.exe (IIRC), so it fails at the configure stage! All of this has cost 
> me quite a few hours in the last week, with very little to show for it.


Hi Andrew,

This has come up quite recently on the MingW lists - see
http://sourceforge.net/mailarchive/forum.php?thread_name=00fc01c7a0af%
240a037c30%240200a8c0%40AMD2500&forum_name=mingw-users for a discussion
of the problem and solution.


Kind regards,

Mark.

-- 
ILande - Open Source Consultancy
http://www.ilande.co.uk

Re: msvc and vista fun

Andrew Dunstan <andrew@dunslane.net> — 2007-06-25T19:28:53Z

Mark Cave-Ayland wrote:
> This has come up quite recently on the MingW lists - see
> http://sourceforge.net/mailarchive/forum.php?thread_name=00fc01c7a0af%
> 240a037c30%240200a8c0%40AMD2500&forum_name=mingw-users for a discussion
> of the problem and solution.
>
>
>
>   

Thanks. I've actually got past this issue, and moved on to the problem 
of having to downgrade far enough to get past the gettimeofday 
redefinition problem :-)

cheers

andrew

Re: [HACKERS] msvc and vista fun

Magnus Hagander <magnus@hagander.net> — 2007-06-25T19:41:05Z

Andrew Dunstan wrote:
> 
> 
> I wrote:
>>
>>
>> Would making a change like this in those 12 places be so ugly?
>>
> 
> Specifically, I propose the following patch, which should fix the issues
> buildfarm apparently has with the XP command shell (or some incarnations
> of it).
> 
> Strictly speaking, we don't need the changes in builddoc.bat and
> install.bat, but I did them for the sake of completeness. Since it
> should not affect anyone not setting XP_EXIT_FIX it seems fairly low risk.

Yeah.
I still would like to have it confirmed that this is actually an XP
fault, and not something else in your environment. To that end, I've
bugged Dave to run some builds on his XP VM (since it apparently works
fine on my XP x64 machine).

//magnus

Re: [HACKERS] msvc and vista fun

Andrew Dunstan <andrew@dunslane.net> — 2007-06-25T19:49:47Z


Magnus Hagander wrote:
> Andrew Dunstan wrote:
>   
>> I wrote:
>>     
>>> Would making a change like this in those 12 places be so ugly?
>>>
>>>       
>> Specifically, I propose the following patch, which should fix the issues
>> buildfarm apparently has with the XP command shell (or some incarnations
>> of it).
>>
>> Strictly speaking, we don't need the changes in builddoc.bat and
>> install.bat, but I did them for the sake of completeness. Since it
>> should not affect anyone not setting XP_EXIT_FIX it seems fairly low risk.
>>     
>
> Yeah.
> I still would like to have it confirmed that this is actually an XP
> fault, and not something else in your environment. To that end, I've
> bugged Dave to run some builds on his XP VM (since it apparently works
> fine on my XP x64 machine).
>
>   

Ok. FWIW, this box is extremely clean and up to date. And I had the 
identical issue on my previous XP/x386 box. It would be very strange if 
somehow two environments got polluted like this.

cheers

andrew

Re: [HACKERS] msvc and vista fun

Magnus Hagander <magnus@hagander.net> — 2007-06-26T11:43:53Z

On Mon, Jun 25, 2007 at 03:19:32PM -0400, Andrew Dunstan wrote:
> 
> 
> I wrote:
> >
> >
> >Would making a change like this in those 12 places be so ugly?
> >
> 
> Specifically, I propose the following patch, which should fix the issues 
> buildfarm apparently has with the XP command shell (or some incarnations 
> of it).

Yeah. Dave has verified taht it's actually because of the XP stuff (he can
repro the failure on XP), so I guess we don't have much choice.

> Strictly speaking, we don't need the changes in builddoc.bat and 
> install.bat, but I did them for the sake of completeness. Since it 
> should not affect anyone not setting XP_EXIT_FIX it seems fairly low risk.

I've applied it, except for builddoc.bat. Since we didn't send out an
exitcode anyway there, it makes no difference. Left them in install.bat
because there's an actual exitcode there.

//Magnus

Re: msvc and vista fun

Andrew Dunstan <andrew@dunslane.net> — 2007-07-24T19:45:43Z


Dave Page wrote:
> Andrew Dunstan wrote:
>
>> On a somewhat related note, I have had spectacular lack of success in 
>> getting either MSVC or MinGW builds to work on Vista - so much so 
>> that I have currently abandoned my attempts on that platform and I 
>> resorted to resuscitating an old XP box for testing. Following some 
>> advice from Magnus, I added ACLs to the build root for both an admin 
>> and a non-admin user (cacls buildroot /E /T /G AdminUser:C and 
>> similarly for a non-admin user) . I can build as the admin user but 
>> when I come to run initdb it fails, complaining that it can't find 
>> the postgres executable. 
>
> Yeah, I ran into that problem as well. I'll look at my Vista box when 
> I'm in the office tomorrow and see if I can figure out what hack fixed 
> it for me.
>
>

I have never heard back on this, AFAIK. If anyone has instructions on 
how to manage this please let me know. My current status with MSVC/vista 
is still that I can build but not run as an admin user, and run but not 
build as a non-admin user. Bleah.

cheers

andrew

Re: msvc and vista fun

Andrei <andyk@commandprompt.com> — 2007-07-24T21:33:32Z

Andrew Dunstan wrote:
> Dave Page wrote:
>> Andrew Dunstan wrote:
>>> On a somewhat related note, I have had spectacular lack of success 
>>> in getting either MSVC or MinGW builds to work on Vista - so much so 
>>> that I have currently abandoned my attempts on that platform and I 
>>> resorted to resuscitating an old XP box for testing. Following some 
>>> advice from Magnus, I added ACLs to the build root for both an admin 
>>> and a non-admin user (cacls buildroot /E /T /G AdminUser:C and 
>>> similarly for a non-admin user) . I can build as the admin user but 
>>> when I come to run initdb it fails, complaining that it can't find 
>>> the postgres executable. 
>> Yeah, I ran into that problem as well. I'll look at my Vista box when 
>> I'm in the office tomorrow and see if I can figure out what hack 
>> fixed it for me.
>
> I have never heard back on this, AFAIK. If anyone has instructions on 
> how to manage this please let me know. My current status with 
> MSVC/vista is still that I can build but not run as an admin user, and 
> run but not build as a non-admin user. Bleah.
>
> cheers
>
> andrew
Described situation looks like you are trying to run initdb under Admin 
account. This will happen even if currently logged user in not admin, 
but initdb has property 'run as administrator' set or you are trying to 
run it under some file commander which is running in admin mode.

Andrei.

Re: msvc and vista fun

Andrew Dunstan <andrew@dunslane.net> — 2007-07-24T21:47:31Z


Andrei Kovalevski wrote:
> Andrew Dunstan wrote:
>> Dave Page wrote:
>>> Andrew Dunstan wrote:
>>>> On a somewhat related note, I have had spectacular lack of success 
>>>> in getting either MSVC or MinGW builds to work on Vista - so much 
>>>> so that I have currently abandoned my attempts on that platform and 
>>>> I resorted to resuscitating an old XP box for testing. Following 
>>>> some advice from Magnus, I added ACLs to the build root for both an 
>>>> admin and a non-admin user (cacls buildroot /E /T /G AdminUser:C 
>>>> and similarly for a non-admin user) . I can build as the admin user 
>>>> but when I come to run initdb it fails, complaining that it can't 
>>>> find the postgres executable. 
>>> Yeah, I ran into that problem as well. I'll look at my Vista box 
>>> when I'm in the office tomorrow and see if I can figure out what 
>>> hack fixed it for me.
>>
>> I have never heard back on this, AFAIK. If anyone has instructions on 
>> how to manage this please let me know. My current status with 
>> MSVC/vista is still that I can build but not run as an admin user, 
>> and run but not build as a non-admin user. Bleah.
>>
>> cheers
>>
>> andrew
> Described situation looks like you are trying to run initdb under 
> Admin account. This will happen even if currently logged user in not 
> admin, but initdb has property 'run as administrator' set or you are 
> trying to run it under some file commander which is running in admin 
> mode.
>
>


No it doesn't. Please read again. As a non-admin user I *can* run. I 
just can't build. As an admin user I can build, but I can't run (and I 
should be able to run).

cheers

andrew

Re: msvc and vista fun

Dave Page <dpage@postgresql.org> — 2007-07-25T08:47:21Z

Andrew Dunstan wrote:
> I have never heard back on this, AFAIK. If anyone has instructions on 
> how to manage this please let me know. My current status with MSVC/vista 
> is still that I can build but not run as an admin user, and run but not 
> build as a non-admin user. Bleah.

I remember responding, but we had 2 somewhat intertwined threads going 
back then as I recall so it might have been in the other one...

On my machine here, I run the buildfarm under user 'Dave' which is a 
member of the Administrators group. It would not run under the 
(unlocked) Administrator account as I recall, as it gave the error you 
report about not being able to find postgres.exe. iirc, this is because 
of the way the administrator account is largely crippled by UAC these 
days (you're not really supposed to use it anyway).

I run the clients for both MSVC and Mingw this way. On Mingw, the only 
other problem I recall was that cc1.exe couldn't be found, for which I 
posted a solution here 
http://archives.postgresql.org/pgsql-hackers/2007-06/msg00932.php

On MSVC, I don't recall seeing the problem you get with mt.exe. Can you 
provide any more useful detail than 'mysterious failure'? That was all 
you said in your original message.

Regards, Dave.