Horton

Thread

  1. some more rambling on the new fe/be communication

    Brett McCormick <brett@work.chicken.org> — 1998-06-01T16:08:09Z

    it would appear that libpq now has a outgoing data buffer associated
with PGconn struct which only gets sent (with send()!?) when pqFlush
gets called.  the backend still appears to use and pass FILE * for
reading and writing.  I wasn't aware that you can read data from a
FILE * sent with send() over a socket.  Is this portable?  Time to
pull out stevens.

In any case, I don't think this bodes well for my SSL patch -- and I
think I've missed something -- why have we switched to send/recv?  I
assume for the synchronous notification?  I haven't been following
that discussion as much as I possibly could be, so I'll look in the
archives.  Anyway, this is sort of a plea for help -- I'm totally
confused, so if there's just something I'm missing, please let me
know.

Does anyone know what implications the new communication scheme has
for SSL?  I know this isn't a postgresql priority, but it is an
interest of mine.  Is it still possible?  I'll start doing my
homework.

  2. Re: [HACKERS] some more rambling on the new fe/be communication

    Tom Lane <tgl@sss.pgh.pa.us> — 1998-06-04T17:23:17Z

    Brett McCormick <brett@work.chicken.org> writes:
> [ Brett was unpleasantly surprised to find major changes in libpq ]

Sorry about that, Brett.  This was discussed on the hackers list a month
or so ago, but evidently you missed the thread.  I made some fairly
major changes in the client-side libpq to allow it to be used
asychronously, that is without blocking until the completion of a query.

I didn't bother to do much cleanup of the backend side, since it didn't
have to change to get the functionality I was after.  I agree that it
could stand a cleanup, so if you want to do it, by all means do.

> it would appear that libpq now has a outgoing data buffer associated
> with PGconn struct which only gets sent (with send()!?) when pqFlush
> gets called.  the backend still appears to use and pass FILE * for
> reading and writing.  I wasn't aware that you can read data from a
> FILE * sent with send() over a socket.  Is this portable?

Yes.  Data on a connection is data; there's no way for the far end to
tell what syscall or library was used to collect and send the data.
(The far end might not even be Unix or C based, after all.)

> In any case, I don't think this bodes well for my SSL patch -- and I
> think I've missed something -- why have we switched to send/recv?

Because going through the stdio library gives up control over blocking
when no data is available.  getc() will block, period.

Does SSL support non-blocking recv?  If so it shouldn't be hard to put
an SSL layer under what I did with libpq.  Note the existence of
PQsocket() however.  If an SSL connection can't be select()'d for then
we have got trouble.

BTW, I believe I did fix your earlier complaint that the backend called
pq_putstr again after closing the client connection.

			regards, tom lane

  3. Re: [HACKERS] some more rambling on the new fe/be communication

    Brett McCormick <brett@work.chicken.org> — 1998-06-04T22:53:08Z

    On Thu, 4 June 1998, at 13:23:17, Tom Lane wrote:

> Sorry about that, Brett.  This was discussed on the hackers list a month
> or so ago, but evidently you missed the thread.  I made some fairly
> major changes in the client-side libpq to allow it to be used
> asychronously, that is without blocking until the completion of a query.

don't be sorry!  that's a good thing.  not sure how I missed that.

> I didn't bother to do much cleanup of the backend side, since it didn't
> have to change to get the functionality I was after.  I agree that it
> could stand a cleanup, so if you want to do it, by all means do.

I beleive I will.  I thought about this last night, and I came up with
this: Since what we need is secure database connections under a stable
release, I'll continue to develop my SSL patch for 6.3.2.  Since I've
already done the work of "cleaning up" (I use quotes because all I've
really done is changed the functions to pass the struct ptr around,
and isolated all read/writes to two functions, pq_read & pq_write)
I'll issue two separate patches.  One which modularizes the IO a
little which will make it easy for people who wish to add other layers
(like kerberos encryption) and an SSL patch to run on top of that.

I'll have to familiarize myself with the new frontend code, but I plan
on making a similar patch for 6.4 (as we'll also want SSL connections
with that).  I am tempted to hold off on that once I get my current
SSL up to snuff and instead work on perl stored procedures, as I feel
that is more valuable (and will also do more to familiarize myself
with the code).

> 
> > it would appear that libpq now has a outgoing data buffer associated
> > with PGconn struct which only gets sent (with send()!?) when pqFlush
> > gets called.  the backend still appears to use and pass FILE * for
> > reading and writing.  I wasn't aware that you can read data from a
> > FILE * sent with send() over a socket.  Is this portable?
> 
> Yes.  Data on a connection is data; there's no way for the far end to
> tell what syscall or library was used to collect and send the data.
> (The far end might not even be Unix or C based, after all.)

What about OOB data?  is that just data as well?

> 
> > In any case, I don't think this bodes well for my SSL patch -- and I
> > think I've missed something -- why have we switched to send/recv?
> 
> Because going through the stdio library gives up control over blocking
> when no data is available.  getc() will block, period.
> 
> Does SSL support non-blocking recv?  If so it shouldn't be hard to put
> an SSL layer under what I did with libpq.  Note the existence of
> PQsocket() however.  If an SSL connection can't be select()'d for then
> we have got trouble.

I'm sure an SSL connection can be select()'d, and it does support
non-blocking recv (I think that's the only way).  I think it does
block, however, if it doesn't get a full SSL "packet" (or whatever the
appropriate term may be).

> BTW, I believe I did fix your earlier complaint that the backend called
> pq_putstr again after closing the client connection.

excellent.

  4. Re: [HACKERS] some more rambling on the new fe/be communication

    Tom Lane <tgl@sss.pgh.pa.us> — 1998-06-04T23:10:09Z

    Brett McCormick <brett@work.chicken.org> writes:
> I'll have to familiarize myself with the new frontend code, but I plan
> on making a similar patch for 6.4 (as we'll also want SSL connections
> with that).

This seems like a reasonable plan, if you need SSL *now* and not after
6.4 is released.  But:

> I am tempted to hold off on that once I get my current
> SSL up to snuff and instead work on perl stored procedures, as I feel
> that is more valuable (and will also do more to familiarize myself
> with the code).

I'd recommend you do the 6.4 version of the patch first, while it's
still fresh in your mind.  AFAIK, stored procedures are a completely
different area of the system; you won't learn anything there that is
relevant to the FE/BE protocol.

> On Thu, 4 June 1998, at 13:23:17, Tom Lane wrote:
>> Yes.  Data on a connection is data; there's no way for the far end to
>> tell what syscall or library was used to collect and send the data.
>> (The far end might not even be Unix or C based, after all.)

> What about OOB data?  is that just data as well?

As far as the TCP protocol is concerned, yes.  A lot of libraries that
you might want to use do not have an API that accounts for the separate
"OOB" channel within one TCP connection, so it may be difficult or
impossible to get at TCP's OOB facility from within a particular
programming environment.  But that's not exactly a cross-environment
compatibility problem, it's just a missing feature in a library API.
Any two implementations that both handle OOB should be able to
communicate.

> I'm sure an SSL connection can be select()'d, and it does support
> non-blocking recv (I think that's the only way).  I think it does
> block, however, if it doesn't get a full SSL "packet" (or whatever the
> appropriate term may be).

Hmm ... I don't know enough about SSL to know if this is really a
problem, but your comment raises warning flags in my head.  This
needs investigation.

			regards, tom lane

  5. Re: [HACKERS] some more rambling on the new fe/be communication

    Brett McCormick <brett@work.chicken.org> — 1998-06-04T23:17:33Z

    On Thu, 4 June 1998, at 19:10:09, Tom Lane wrote:

> > I am tempted to hold off on that once I get my current
> > SSL up to snuff and instead work on perl stored procedures, as I feel
> > that is more valuable (and will also do more to familiarize myself
> > with the code).
> 
> I'd recommend you do the 6.4 version of the patch first, while it's
> still fresh in your mind.  AFAIK, stored procedures are a completely
> different area of the system; you won't learn anything there that is
> relevant to the FE/BE protocol.

I know -- I'm looking to learn more about other areas of the system.
And because the perl stored procedures will support lots of cool
backend functions (what sort of stuff is permissible to interface to?)
i'll learn about them that way.

I've learned as much about the fe/be protocol as I wish to know ;)
But I may take your advice.

> > I'm sure an SSL connection can be select()'d, and it does support
> > non-blocking recv (I think that's the only way).  I think it does
> > block, however, if it doesn't get a full SSL "packet" (or whatever the
> > appropriate term may be).
> 
> Hmm ... I don't know enough about SSL to know if this is really a
> problem, but your comment raises warning flags in my head.  This
> needs investigation.

Unfortunately, SSL documention is rather incomplete.

thanks for your info, comments and suggestions

  6. Re: [HACKERS] some more rambling on the new fe/be communication

    Bruce Momjian <maillist@candle.pha.pa.us> — 1998-06-04T23:46:09Z

    > As far as the TCP protocol is concerned, yes.  A lot of libraries that
> you might want to use do not have an API that accounts for the separate
> "OOB" channel within one TCP connection, so it may be difficult or
> impossible to get at TCP's OOB facility from within a particular
> programming environment.  But that's not exactly a cross-environment
> compatibility problem, it's just a missing feature in a library API.
> Any two implementations that both handle OOB should be able to
> communicate.

Looks like we will be removing OOB in favor of a CANCEL cookie sent to
the postmaster.  I will work up something soon.

-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)