Horton

Thread

  1. Regression failure with PostgreSQL 8beta1 and Intel Itanium 2 C compiler

    Robert E. Bruccoleri <bruc@stone.congenomics.com> — 2004-08-10T23:12:22Z

    Dear All,
	I built PG 8.0 beta1 on an Itanium 2 platform using the Intel compilers
version 8, and got one real difference in the regression tests that affected
int2, int4, union, and numerology. Here's the key difference:

horta postgres 177 > diff -c int4.out ../expected/
*** int4.out    Tue Aug 10 18:41:48 2004
--- ../expected/int4.out        Wed Mar 10 21:11:13 2004
***************
*** 22,27 ****
--- 22,28 ----
  INSERT INTO INT4_TBL(f1) VALUES ('   asdf   ');
  ERROR:  invalid input syntax for integer: "   asdf   "
  INSERT INTO INT4_TBL(f1) VALUES ('- 1234');
+ ERROR:  invalid input syntax for integer: "- 1234"
  INSERT INTO INT4_TBL(f1) VALUES ('123       5');
  ERROR:  invalid input syntax for integer: "123       5"
  INSERT INTO INT4_TBL(f1) VALUES ('');

PG 8.0 beta1 is accepting "- 1234" as a valid integer. Further investigation
reveals that this is a peculiarity of the Intel compilers. The following
program,

#include <stdio.h>
#include <stdlib.h>

main() {
  char st[] = "- 1234";
  int l;
  char *endp;

  l = strtol(st, &endp, 10);
  printf("l = %d  st = %lx  endp = %lx\n", l, st, endp);
}

using the Intel compiler provided libraries prints

l = -1234  st = 60000fffffffb720  endp = 60000fffffffb726

whereas gcc and glibc yields

l = 0  st = 60000fffffffb710  endp = 60000fffffffb710

Boo hiss...

+-----------------------------+------------------------------------+
| Robert E. Bruccoleri, Ph.D. | email: bruc@acm.org                |
| President, Congenair LLC    | URL:   http://www.congen.com/~bruc |
| P.O. Box 314                | Phone: 609 818 7251                | 
| Pennington, NJ 08534        |                                    |
+-----------------------------+------------------------------------+

  2. Re: Regression failure with PostgreSQL 8beta1 and Intel

    Hans-Jürgen Schönig <postgres@cybertec.at> — 2004-08-12T06:52:53Z

    Robert,

Are you planning to use Intel's C compiler in production?
We tried that some time ago and corrupted our database cluster almost 
instantly (for some reason we have not investigated any further).
I highly recommend to do some stress testing to see if everything works 
nicely.
I'd be pleased to get some feedback.

	Regard,

		Hans



Robert E. Bruccoleri wrote:
> Dear All,
> 	I built PG 8.0 beta1 on an Itanium 2 platform using the Intel compilers
> version 8, and got one real difference in the regression tests that affected
> int2, int4, union, and numerology. Here's the key difference:
> 
> horta postgres 177 > diff -c int4.out ../expected/
> *** int4.out    Tue Aug 10 18:41:48 2004
> --- ../expected/int4.out        Wed Mar 10 21:11:13 2004
> ***************
> *** 22,27 ****
> --- 22,28 ----
>   INSERT INTO INT4_TBL(f1) VALUES ('   asdf   ');
>   ERROR:  invalid input syntax for integer: "   asdf   "
>   INSERT INTO INT4_TBL(f1) VALUES ('- 1234');
> + ERROR:  invalid input syntax for integer: "- 1234"
>   INSERT INTO INT4_TBL(f1) VALUES ('123       5');
>   ERROR:  invalid input syntax for integer: "123       5"
>   INSERT INTO INT4_TBL(f1) VALUES ('');
> 
> PG 8.0 beta1 is accepting "- 1234" as a valid integer. Further investigation
> reveals that this is a peculiarity of the Intel compilers. The following
> program,
> 
> #include <stdio.h>
> #include <stdlib.h>
> 
> main() {
>   char st[] = "- 1234";
>   int l;
>   char *endp;
> 
>   l = strtol(st, &endp, 10);
>   printf("l = %d  st = %lx  endp = %lx\n", l, st, endp);
> }
> 
> using the Intel compiler provided libraries prints
> 
> l = -1234  st = 60000fffffffb720  endp = 60000fffffffb726
> 
> whereas gcc and glibc yields
> 
> l = 0  st = 60000fffffffb710  endp = 60000fffffffb710
> 
> Boo hiss...
> 
> +-----------------------------+------------------------------------+
> | Robert E. Bruccoleri, Ph.D. | email: bruc@acm.org                |
> | President, Congenair LLC    | URL:   http://www.congen.com/~bruc |
> | P.O. Box 314                | Phone: 609 818 7251                | 
> | Pennington, NJ 08534        |                                    |
> +-----------------------------+------------------------------------+
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 8: explain analyze is your friend


-- 
Cybertec Geschwinde u Schoenig
Schoengrabern 134, A-2020 Hollabrunn, Austria
Tel: +43/720/10 1234567 or +43/660/816 40 77
www.cybertec.at, www.postgresql.at, kernel.cybertec.at

  3. Turkish downcasting in PL/pgSQL

    ntufar <ntufar@pisem.net> — 2004-08-12T12:26:57Z

    Your name               :
Your email address      :


System Configuration
---------------------
  Architecture (example: Intel Pentium)         : Intel Pentium

  Operating System (example: Linux 2.4.18)      : Debian unstable 
						  Linux 2.6.6-1-k7

  PostgreSQL version (example: PostgreSQL-8.0):  PostgreSQL-8.0 CVS HEAD

  Compiler used (example:  gcc 2.95.2)          : gcc 3.3.4


Please enter a FULL description of your problem:
------------------------------------------------

Problems with Turkish locale are widely known to developers. 
Another one, now in PL/pgSQL have reared it's ugly head.
Regression tests are failing at triggers, plpgsql, copy2 
and rangefuncs. Examienation of regression.diff showed that
the failures were due to unrecognised statements like 
BEGIN, RAISE and IF in PL/pgSQL functions. Replacing
capital "I" with lower-case "i" (BEGiN, RAiSE, iF) completely
sloves the problem.


If you know how this problem might be fixed, list the solution below:
---------------------------------------------------------------------

Apparently problem is caused by the following directive:

     %option case-insensitive

on line 76 in file src/pl/plpgsql/src/scan.l

flex (flex version 2.5.4) incorporates case-insensitivity in it's 
state tables because if I run flex stage with LANG=C everything 
works fine. A quick and dirty fix could be implemented by placing

     LANG=C
     export LANG

in file src/pl/plpgsql/src/Makefile before calling flex.

A long term fix can be done by implementing a function
for keyword lookup like ScanKeywordLookup() in
src/backend/parser/keywords.c.

I would gladly prepare a patch and send it for your consideration 
tomorrow morning.

Best regards,
Nicolai Tufar

  4. Re: Turkish downcasting in PL/pgSQL

    Tom Lane <tgl@sss.pgh.pa.us> — 2004-08-12T15:32:41Z

    ntufar <ntufar@pisem.net> writes:
> flex (flex version 2.5.4) incorporates case-insensitivity in it's 
> state tables because if I run flex stage with LANG=C everything 
> works fine.

Ick.  That is of course why it worked for me when I tested it :-(

> A quick and dirty fix could be implemented by placing
>      LANG=C
>      export LANG
> in file src/pl/plpgsql/src/Makefile before calling flex.

This is probably what we'd better do.  Otherwise we have
build-context-dependency in the system's behavior, which is bad.

Peter, any thoughts on this one way or the other?  At the moment
plpgsql's scan.l seems to be the only use of '%option case-insensitive'
but we have enough flex lexers laying about that I wouldn't be surprised
to have this same risk elsewhere.  Is it reasonable to try to force
LANG=C in some global fashion during the build?

			regards, tom lane

  5. Re: Turkish downcasting in PL/pgSQL

    ntufar <ntufar@pisem.net> — 2004-08-12T19:09:11Z

    Greetings,


12-08-2004 Perşembe günü saat 18:32 sularında, Tom Lane dedi ki:
ntufar <ntufar@pisem.net> writes:
> > flex (flex version 2.5.4) incorporates case-insensitivity in it's
> > state tables because if I run flex stage with LANG=C everything
> > works fine.
>
> Ick.  That is of course why it worked for me when I tested it :-(
>
> > A quick and dirty fix could be implemented by placing
> >      LANG=C
> >      export LANG
> > in file src/pl/plpgsql/src/Makefile before calling flex.
>
> This is probably what we'd better do.  Otherwise we have
> build-context-dependency in the system's behavior, which is bad.
>
I attached a diff of fix that adds LANG=C; before call to $(FLEX).
Fixes the problem here but I don't know if adding environment variable
assignment like this is appropriate. I am not too fluent in PostgreSQL
build environment and do not know where one can put a global deffinition
you are talking below.

Peter, any thoughts on this one way or the other?  At the moment
> plpgsql's scan.l seems to be the only use of '%option
case-insensitive'
> but we have enough flex lexers laying about that I wouldn't be
surprised
> to have this same risk elsewhere.  Is it reasonable to try to force
> LANG=C in some global fashion during the build?
>
>                       regards, tom lane
>
Best regards,
Nicolai Tufar

  6. Re: Turkish downcasting in PL/pgSQL

    Tom Lane <tgl@sss.pgh.pa.us> — 2004-08-12T19:27:29Z

    ntufar <ntufar@pisem.net> writes:
> I attached a diff of fix that adds LANG=C; before call to $(FLEX).
> Fixes the problem here but I don't know if adding environment variable
> assignment like this is appropriate. I am not too fluent in PostgreSQL
> build environment and do not know where one can put a global deffinition
> you are talking below.

Um, the attachment was unreadable :-( but I get the idea.

As for the global solution, I was wondering if it would work to put
"LANG=C" right inside the definition of $(FLEX).  That would ensure
the right behavior from all our flex builds without unnecessarily
messing up people's build environments otherwise.  I don't know however
whether this would parse properly.

			regards, tom lane

  7. Re: Regression failure with PostgreSQL 8beta1 and Intel

    Robert E. Bruccoleri <bruc@stone.congenomics.com> — 2004-08-12T19:30:53Z

    Dear Hans,
> 
> Robert,
> 
> Are you planning to use Intel's C compiler in production?
> We tried that some time ago and corrupted our database cluster almost 
> instantly (for some reason we have not investigated any further).
> I highly recommend to do some stress testing to see if everything works 
> nicely.
> I'd be pleased to get some feedback.

We're using the Intel C compiler for all our code on the Altix. We
have encountered a few bugs with optimization, but on the whole, it
works pretty well, and it generates code that runs about 30% faster
than gcc (depending on the application of course). With regard to
PostgreSQL, we are not having massive problems with data
corruption. We have had problems with 7.4.3, and I'm currently testing
8.0.0beta1. I don't believe the problem with 7.4.3 is a compiler issue --
it only happens when we have multiple backends running at the same time.
Other people like Hubert Froehlich
(http://archives.postgresql.org/pgsql-general/2004-07/msg00670.php)
have run into the same problem using a completely different environment.
I'll report on 8.0.0beta1 when our testing is complete. --Bob

+-----------------------------+------------------------------------+
| Robert E. Bruccoleri, Ph.D. | email: bruc@acm.org                |
| President, Congenair LLC    | URL:   http://www.congen.com/~bruc |
| P.O. Box 314                | Phone: 609 818 7251                | 
| Pennington, NJ 08534        |                                    |
+-----------------------------+------------------------------------+

  8. Re: Turkish downcasting in PL/pgSQL

    ntufar <ntufar@pisem.net> — 2004-08-12T19:47:54Z

    12-08-2004 Perşembe günü saat 22:27 sularında, Tom Lane dedi ki:
> ntufar <ntufar@pisem.net> writes:
> > I attached a diff of fix that adds LANG=C; before call to $(FLEX).
> > Fixes the problem here but I don't know if adding environment variable
> > assignment like this is appropriate. I am not too fluent in PostgreSQL
> > build environment and do not know where one can put a global deffinition
> > you are talking below.
> 
> Um, the attachment was unreadable :-( but I get the idea.

Something to do with my mail provider, sorry.
in file src/pl/plpgsql/src/Makefile:
	LANG=C;$(FLEX) $(FLEXFLAGS) -Pplpgsql_base_yy -o'$@' $<
instead of
	$(FLEX) $(FLEXFLAGS) -Pplpgsql_base_yy -o'$@' $<

> 
> As for the global solution, I was wondering if it would work to put
> "LANG=C" right inside the definition of $(FLEX).  That would ensure
> the right behavior from all our flex builds without unnecessarily
> messing up people's build environments otherwise.  I don't know however
> whether this would parse properly.

The only thing that comest in mind is that it may break Win32 port. 
Can someone comment on this?

> 
> 			regards, tom lane

Regards,
Nicolai Tufar

  9. Re: Turkish downcasting in PL/pgSQL

    Peter Eisentraut <peter_e@gmx.net> — 2004-08-14T08:21:33Z

    Tom Lane wrote:
> Peter, any thoughts on this one way or the other?  At the moment
> plpgsql's scan.l seems to be the only use of '%option
> case-insensitive' but we have enough flex lexers laying about that I
> wouldn't be surprised to have this same risk elsewhere.  Is it
> reasonable to try to force LANG=C in some global fashion during the
> build?

You'd have to set LC_ALL=C to be really sure to override everything.  
But I would stay away from doing that globally, because all the 
translation work in gcc and make would go to waste.

I would also suggest that Nicolai report this issue to the flex 
developers.  It's only bound to reappear everywhere case-insensitive 
flex scanners are used.

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

  10. Re: Turkish downcasting in PL/pgSQL

    Peter Eisentraut <peter_e@gmx.net> — 2004-08-14T08:34:38Z

    ntufar wrote:
> Apparently problem is caused by the following directive:
>
>      %option case-insensitive
>
> on line 76 in file src/pl/plpgsql/src/scan.l
>
> flex (flex version 2.5.4) incorporates case-insensitivity in it's
> state tables because if I run flex stage with LANG=C everything
> works fine. A quick and dirty fix could be implemented by placing
>
>      LANG=C
>      export LANG
>
> in file src/pl/plpgsql/src/Makefile before calling flex.

I have tried running flex (2.5.4) with a number of different locales 
including tr_TR, but the output file is always the same.  Can you show 
us a diff of the generated files?

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

  11. Re: Turkish downcasting in PL/pgSQL

    Tom Lane <tgl@sss.pgh.pa.us> — 2004-08-14T15:21:46Z

    Peter Eisentraut <peter_e@gmx.net> writes:
> You'd have to set LC_ALL=C to be really sure to override everything.  
> But I would stay away from doing that globally, because all the 
> translation work in gcc and make would go to waste.

Agreed.  I was toying with changing the FLEX variable to contain
"LC_ALL=C flex" but I'm a bit worried about breaking the build on
some platforms (especially Windows).

> I would also suggest that Nicolai report this issue to the flex 
> developers.  It's only bound to reappear everywhere case-insensitive 
> flex scanners are used.

True.  Maybe we should just call it a flex bug and wait for them to
fix it.  It's not going to affect builds from tarballs anyway, only
people who build from CVS.

			regards, tom lane

  12. Re: Turkish downcasting in PL/pgSQL

    Tom Lane <tgl@sss.pgh.pa.us> — 2004-08-14T16:03:16Z

    Peter Eisentraut <peter_e@gmx.net> writes:
> I have tried running flex (2.5.4) with a number of different locales 
> including tr_TR, but the output file is always the same.  Can you show 
> us a diff of the generated files?

Hmm ... a quick look at the flex sources shows that flex does rely on
the <ctype.h> routines for case-folding, so I have no doubt that
ntufar's report is accurate.  Maybe you used the wrong tr_TR locale?

(Just for the record, though, I can't see any change in the generated
pl_scan.c output in any of the tr_TR variants available on either HPUX
or OS X.  I don't have a full set of locales installed on my Linux
machine so I can't try it there.)

			regards, tom lane

  13. Re: Turkish downcasting in PL/pgSQL

    Devrim GÜNDÜZ <devrim@gunduz.org> — 2004-08-16T11:16:47Z

    Hi,

On Thu, 12 Aug 2004, Tom Lane wrote:

> > flex (flex version 2.5.4) incorporates case-insensitivity in it's 
> > state tables because if I run flex stage with LANG=C everything 
> > works fine.
> 
> Ick.  That is of course why it worked for me when I tested it :-(

Nicolai is on holiday now. I tested on my Fedora Core 2 and RHEL 3 ES 
systems and all regression tests passed:

======================
 All 96 tests passed.
======================

I'm using the latest tr_TR locale of glibc, and flex-2.5.4a-29 (of RHEL) 
and flex-2.5.4a-31 (of FC 2).

What am I missing?

Regards,
-- 
Devrim GUNDUZ	       
devrim~gunduz.org				devrim.gunduz~linux.org.tr 
			http://www.tdmsoft.com
			http://www.gunduz.org

  14. Re: Turkish downcasting in PL/pgSQL

    Tom Lane <tgl@sss.pgh.pa.us> — 2004-08-16T14:05:11Z

    Devrim GUNDUZ <devrim@gunduz.org> writes:
>  All 96 tests passed.

> I'm using the latest tr_TR locale of glibc, and flex-2.5.4a-29 (of RHEL) 
> and flex-2.5.4a-31 (of FC 2).

> What am I missing?

If you built from a tarball, then the flex run is already done for you.
Remove src/pl/plpgsql/src/pl_scan.c and rebuild to see if you see a
problem.

			regards, tom lane

  15. Re: Turkish downcasting in PL/pgSQL

    Devrim GÜNDÜZ <devrim@gunduz.org> — 2004-08-16T14:27:29Z

    -----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

On Mon, 16 Aug 2004, Tom Lane wrote:

> > I'm using the latest tr_TR locale of glibc, and flex-2.5.4a-29 (of RHEL) 
> > and flex-2.5.4a-31 (of FC 2).
> 
> > What am I missing?
> 
> If you built from a tarball, then the flex run is already done for you.
> Remove src/pl/plpgsql/src/pl_scan.c and rebuild to see if you see a
> problem.

I tried beta1 and latest CVS snapshot before sending the mail. All 
produced the same result.

A few minutes before I tried on a Debian unstable, as Nicolai reported. 
But all the regression tests passed again, using the latest flex + 
glibc... I can't reproduce the problem :( Or there is not a bug :)

Regards,
- -- 
Devrim GUNDUZ	       
devrim~gunduz.org				devrim.gunduz~linux.org.tr 
			http://www.tdmsoft.com
			http://www.gunduz.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFBIMRTtl86P3SPfQ4RAhPVAJ9g+UZRyv6SROp9TOzzp2/UsD/W4gCfSKNI
3cdoGjAZ4WLLZHXWs0Wq5Lk=
=1TsT
-----END PGP SIGNATURE-----

  16. Re: Turkish downcasting in PL/pgSQL

    Tom Lane <tgl@sss.pgh.pa.us> — 2004-08-16T14:56:44Z

    Devrim GUNDUZ <devrim@gunduz.org> writes:
> A few minutes before I tried on a Debian unstable, as Nicolai reported. 
> But all the regression tests passed again, using the latest flex + 
> glibc... I can't reproduce the problem :( Or there is not a bug :)

Hmph.  Either Nicolai has a weird locale setting, or he made a mistake.

We'll have to put this on hold until he gets back, I guess.  Fortunately
there's still lots of time till release.

			regards, tom lane