Thread

1. psql, remove include of psqlscan.c

   Karl O. Pinc <kop@meme.com> — 2012-09-27T15:00:02Z

   Hi,
   
   This patch (psql_remove_include.patch) eliminates 
   the #include of psqlscan.c at the bottom of mainloop.c.
   
   The attached patch uses the %top{} flex feature 
   introduced in flex 2.5.30 released 2003-4-1.
   (See the NEWS file for flex.)
   
   The good news is that config/programs.m4
   requires flex >= 2.5.31.  The bad news 
   is that RHEL 5 (released 2007-03-14
   with a 13 year (10+3 years) support lifecycle)
   has a flex (2.5.4a) that is too old for this patch. :-(
   (At least this is what the changelog in flex
   from the RHEL 5 srpm repo tells me.)
   I don't know what this means.
   
   The flex docs on my box, fwiw, don't mention %top{}
   in the section describing Posix (in)compatibility.
   
   The patch is against git head.  All the tests
   pass, fwiw.  Because this depends on the toolchain
   it wouldn't hurt to build on other architectures/
   operating systems.
   
   I'm thinking of exposing enough of the psql parser,
   moving it to libpq, that any client-side app can
   do what libpq does; given a bunch of sql
   separated by semi-colons get the results
   of all the statements.  This should also allow
   the "statement separation" to be done on the client
   side in libpq.  Although I don't imagine that this
   will have a performance impact on the server side
   it sounds like a first step toward pushing more of
   the parsing onto the client.
   
   Regards,
   
   Karl <kop@meme.com>
   Free Software:  "You don't pay back, you pay forward."
                    -- Robert A. Heinlein
   
   P.S.  Sent this patch to the list at Sep 27 00:43:28 Central time
   and it never showed up so I'm sending again with a different
   subject.  Info on the previous message:
   
   Subject:	Remove #include psqlscan.c from psql
   message-id=<1348724599.28442.0@mofo>
   
   

2. Re: psql, remove include of psqlscan.c

   Tom Lane <tgl@sss.pgh.pa.us> — 2012-09-27T15:07:48Z

   "Karl O. Pinc" <kop@meme.com> writes:
   > This patch (psql_remove_include.patch) eliminates 
   > the #include of psqlscan.c at the bottom of mainloop.c.
   
   I don't really see that this is enough of an improvement to justify
   depending on a non-portable flex feature.
   
   > I'm thinking of exposing enough of the psql parser,
   > moving it to libpq, that any client-side app can
   > do what libpq does; given a bunch of sql
   > separated by semi-colons get the results
   > of all the statements.  This should also allow
   > the "statement separation" to be done on the client
   > side in libpq.  Although I don't imagine that this
   > will have a performance impact on the server side
   > it sounds like a first step toward pushing more of
   > the parsing onto the client.
   
   Quite honestly, I don't like that idea at all either.  It's bad enough
   that psql has to know so much low-level SQL syntax.  If we start
   encouraging other programs to know that, we're going to be locked into
   never again making any lexical-level changes.  Why exactly is "pushing
   parsing onto the client" a good idea?
   
   			regards, tom lane
   
   
   

3. Re: psql, remove include of psqlscan.c

   Karl O. Pinc <kop@meme.com> — 2012-09-27T15:29:53Z

   On 09/27/2012 10:07:48 AM, Tom Lane wrote:
   > "Karl O. Pinc" <kop@meme.com> writes:
   > > This patch (psql_remove_include.patch) eliminates 
   > > the #include of psqlscan.c at the bottom of mainloop.c.
   > 
   > I don't really see that this is enough of an improvement to justify
   > depending on a non-portable flex feature.
   
   Ok.  (I had no idea it was non-portable.  The flex
   docs don't mention this.)
   
   > 
   > > I'm thinking of exposing enough of the psql parser,
   > > moving it to libpq, that any client-side app can
   > > do what libpq does; given a bunch of sql
   > > separated by semi-colons get the results
   > > of all the statements.  This should also allow
   > > the "statement separation" to be done on the client
   > > side in libpq.  Although I don't imagine that this
   > > will have a performance impact on the server side
   > > it sounds like a first step toward pushing more of
   > > the parsing onto the client.
   > 
   > Quite honestly, I don't like that idea at all either.  It's bad 
   > enough
   > that psql has to know so much low-level SQL syntax.  If we start
   > encouraging other programs to know that, we're going to be locked 
   > into
   > never again making any lexical-level changes.  Why exactly is 
   > "pushing
   > parsing onto the client" a good idea?
   
   There's really 2 ideas.  I don't care about this one:
   pushing parsing/anything onto the client is a good idea because
   clients scale horizontally and so performance is improved.
   
   What I'm thinking of in libpq is the ability to give it big string
   with many sql statements and have it hand back each statement
   so the client can then submit it to the server for execution.
   What I really _want_ is to be able get a bit string of many
   sql statements from the user and return the results, statuses,
   etc. of executing each statement.  Just what psql does when,
   say, fed a file from stdin.
   
   The reason I want this is because I don't want to have to
   rewrite the sql parser in PHP for inclusion in phpPgAdmin.
   (I did this once, and it was such a big ugly patch
   it never got around to getting into the mainline phpPgAdmin.)
   phpPgAdmin (pgAdmin/ front-end of your choice)
   should be able process a string of sql statements in the
   same way that psql does, but currently the only way to
   do this is to rewrite the parser in each application.
   Much better to have libpq provide the functionality,
   although I suppose it's possible to push this into the
   server.
   
   
   Karl <kop@meme.com>
   Free Software:  "You don't pay back, you pay forward."
                    -- Robert A. Heinlein
   
   
   
   

4. Re: psql, remove include of psqlscan.c

   Alvaro Herrera <alvherre@2ndquadrant.com> — 2012-09-27T15:41:35Z

   Excerpts from Karl O. Pinc's message of jue sep 27 12:29:53 -0300 2012:
   
   > The reason I want this is because I don't want to have to
   > rewrite the sql parser in PHP for inclusion in phpPgAdmin.
   > (I did this once, and it was such a big ugly patch
   > it never got around to getting into the mainline phpPgAdmin.)
   > phpPgAdmin (pgAdmin/ front-end of your choice)
   > should be able process a string of sql statements in the
   > same way that psql does, but currently the only way to
   > do this is to rewrite the parser in each application.
   > Much better to have libpq provide the functionality,
   > although I suppose it's possible to push this into the
   > server.
   
   This seems a worthy goal to me.
   
   But I think I see what Tom objection to it is: if we "export" this
   capability to libpq applications, then we set it in stone to a certain
   extent: exactly how things are split would become part of the API, so to
   speak.  Upgrading to a newer libpq could break application code that
   worked with the previous release's by splitting things differently.
   
   I don't currently have an opinion on whether this is a bad thing or not.
   Who wants to argue for/against?
   
   -- 
   Álvaro Herrera                http://www.2ndQuadrant.com/
   PostgreSQL Development, 24x7 Support, Training & Services
   
   
   

5. Re: psql, remove include of psqlscan.c

   John R Pierce <pierce@hogranch.com> — 2012-09-27T15:48:38Z

   On 09/27/12 8:41 AM, Alvaro Herrera wrote:
   > But I think I see what Tom objection to it is: if we "export" this
   > capability to libpq applications, then we set it in stone to a certain
   > extent: exactly how things are split would become part of the API, so to
   > speak.  Upgrading to a newer libpq could break application code that
   > worked with the previous release's by splitting things differently.
   >
   > I don't currently have an opinion on whether this is a bad thing or not.
   > Who wants to argue for/against?
   
   I wonder if it shouldn't be in a separate 'helper' library, as it has no 
   direct ties to any libpq internals.
   
   
   
   -- 
   john r pierce                            N 37, W 122
   santa cruz ca                         mid-left coast
   
   
   
   

6. Re: psql, remove include of psqlscan.c

   Tom Lane <tgl@sss.pgh.pa.us> — 2012-09-27T16:02:42Z

   Alvaro Herrera <alvherre@2ndquadrant.com> writes:
   > But I think I see what Tom objection to it is: if we "export" this
   > capability to libpq applications, then we set it in stone to a certain
   > extent: exactly how things are split would become part of the API, so to
   > speak.  Upgrading to a newer libpq could break application code that
   > worked with the previous release's by splitting things differently.
   
   That's not exactly what the problem is: an upgrade could only break
   *existing* application code if we had made a non-backward-compatible
   change in SQL lexical rules, which hopefully we'd never do.
   
   Rather, the problem is that the server might know about some newer
   lexical feature, and so might the application, but if libpq is behind
   the times then it's broken.  You can see an instance of this right now
   over in the pgsql-jdbc list, where they're arguing about fixing the
   JDBC driver to understand dollar quoting.  Application authors not
   unreasonably think it should have heard of that by now.
   
   The JDBC example is sort of an argument in favor of Karl's idea,
   in that if client-side code is going to know this anyway then it would
   be better if it were at least localized in one place.  But of course
   JDBC is never going to depend on libpq (wrong language) so moving this
   code into libpq isn't actually going to help them.
   
   A larger point is that I don't believe this is actually going to help
   anybody, because of mismatch of requirements not only implementation
   language.  JDBC couldn't use a libpq lexer implementation even without
   the language issue, because the context in which they're arguing about
   this is finding and replacing JDBC-spec escape sequences, which libpq is
   not going to know about.  I imagine PHP has got the same problem only
   different.  Conversely, psql's lexer has a lot of psql-specific design
   decisions, such as the need to handle backslash commands and include
   files, that I don't think would belong in a libpq implementation.
   
   			regards, tom lane
   
   
   

7. Re: psql, remove include of psqlscan.c

   Karl O. Pinc <kop@meme.com> — 2012-09-27T18:02:14Z

   On 09/27/2012 11:02:42 AM, Tom Lane wrote:
   > Alvaro Herrera <alvherre@2ndquadrant.com> writes:
   
   > A larger point is that I don't believe this is actually going to help
   > anybody, because of mismatch of requirements not only implementation
   > language.  JDBC couldn't use a libpq lexer implementation even 
   > without
   > the language issue, because the context in which they're arguing 
   > about
   > this is finding and replacing JDBC-spec escape sequences, which libpq
   > is
   > not going to know about.  I imagine PHP has got the same problem only
   > different.  Conversely, psql's lexer has a lot of psql-specific 
   > design
   > decisions, such as the need to handle backslash commands and include
   > files, that I don't think would belong in a libpq implementation.
   
   Well no, I'm not at all interested in escape sequences.
   I want to take sql directly from the user and execute it.
   Right now I can take only one statement at a time.
   And this is true of any human-facing application.
   
   I'm not looking for a general purpose solution, although
   it did occur to me that the psql variable substitution
   mechanism could be exposed.
   
   But what I really want is not "an exposed parser".  What
   I really want is to be able to get results from all the
   statements passed to PQexec (et-al), not just the
   last statement.  This could be done without exposing
   the parser, but it does mean having a parser in libpq.
   Since psql links to libpq anyway my plan was to
   move the parser entirely out of psql into libpq and
   have an undocumented internal interface so that
   psql can do the escaping/variable substitution stuff.
   
   Regards,
   
   Karl <kop@meme.com>
   Free Software:  "You don't pay back, you pay forward."
                    -- Robert A. Heinlein
   
   
   
   

8. Re: psql, remove include of psqlscan.c

   Karl O. Pinc <kop@meme.com> — 2012-09-27T18:06:34Z

   On 09/27/2012 11:02:42 AM, Tom Lane wrote:
   
   > Rather, the problem is that the server might know about some newer
   > lexical feature, and so might the application, but if libpq is behind
   > the times then it's broken. 
   
   If the application knows about the newer feature and wants
   to use it, is it unreasonable to ask the application
   be linked against a newer libpq?
   
   
   Karl <kop@meme.com>
   Free Software:  "You don't pay back, you pay forward."
                    -- Robert A. Heinlein
   
   
   
   

9. Re: psql, remove include of psqlscan.c

   Tom Lane <tgl@sss.pgh.pa.us> — 2012-09-27T19:28:49Z

   "Karl O. Pinc" <kop@meme.com> writes:
   > What I'm thinking of in libpq is the ability to give it big string
   > with many sql statements and have it hand back each statement
   > so the client can then submit it to the server for execution.
   > What I really _want_ is to be able get a bit string of many
   > sql statements from the user and return the results, statuses,
   > etc. of executing each statement.  Just what psql does when,
   > say, fed a file from stdin.
   
   Just as a note --- I believe you can get that result today with
   PQsendQuery followed by a PQgetResult loop.  There's no need to
   provide an API that splits the string first.
   
   			regards, tom lane
   
   
   

10. Re: psql, remove include of psqlscan.c

    Tom Lane <tgl@sss.pgh.pa.us> — 2012-09-27T19:37:00Z

    "Karl O. Pinc" <kop@meme.com> writes:
    > On 09/27/2012 11:02:42 AM, Tom Lane wrote:
    >> Rather, the problem is that the server might know about some newer
    >> lexical feature, and so might the application, but if libpq is behind
    >> the times then it's broken. 
    
    > If the application knows about the newer feature and wants
    > to use it, is it unreasonable to ask the application
    > be linked against a newer libpq?
    
    Well, see the JDBC example: there is no newer driver with the desired
    feature for applications to link against.  Even if all client-side code
    with such knowledge stays perfectly in sync as far as upstream sources
    are concerned, there are any number of reasons why particular
    installations might have copies of varying vintages.  It's not really
    a situation that I want to get into more than necessary.
    
    			regards, tom lane
    
    
    

11. Re: psql, remove include of psqlscan.c

    Karl O. Pinc <kop@meme.com> — 2012-09-27T20:06:16Z

    On 09/27/2012 02:28:49 PM, Tom Lane wrote:
    > "Karl O. Pinc" <kop@meme.com> writes:
    
    > > What I really _want_ is to be able get a bit string of many
    > > sql statements from the user and return the results, statuses,
    > > etc. of executing each statement.  Just what psql does when,
    > > say, fed a file from stdin.
    > 
    > Just as a note --- I believe you can get that result today with
    > PQsendQuery followed by a PQgetResult loop.  There's no need to
    > provide an API that splits the string first.
    
    Ah.  *sigh*  I should pay better attention.  Sorry for the
    distraction.
    
    
    Karl <kop@meme.com>
    Free Software:  "You don't pay back, you pay forward."
                     -- Robert A. Heinlein