Horton

Thread

Commits

  1. Add DNS SRV support for LDAP server discovery.

  1. DNS SRV support for LDAP authentication

    Thomas Munro <thomas.munro@enterprisedb.com> — 2018-09-25T02:09:08Z

    Hello hackers,

Some people like to use DNS SRV records to advertise LDAP servers on
their network.  Microsoft Active Directory is usually (always?) set up
that way.  Here is a patch to allow our LDAP auth module to support
that kind of discovery.  It copies the convention of the OpenLDAP
command line tools: if you give it a URL that has no hostname, it'll
try to extract a domain name from the bind DN, and then ask your DNS
server for a SRV record for LDAP-over-TCP at that domain.  The
OpenLDAP version of libldap.so exports the magic to do that, so the
patch is very small (but the infrastructure set-up to test it is a bit
of a schlep, see below).  I'll add this to the next Commitfest.

Testing instructions for (paths and commands given for FreeBSD, adjust
as appropriate):

1.  Install BIND:

$ sudo pkg install bind99

2.  Define a new zone for testing, by adding the following to the end
of /usr/local/etc/namedb/named.conf:

===== 8< =====
zone "my.test.domain" {
        type master;
        file "/usr/local/etc/namedb/master/my.test.domain";
};
===== 8< =====


3.  Create that zone file in /usr/local/etc/namedb/master/my.test.domain:

===== 8< =====
$TTL    10
@       IN      SOA     ns.my.test.domain. admin.my.test.domain. (
                              2         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
        IN      NS      ns.my.test.domain.
ns.my.test.domain.      IN      A       127.0.0.1
my.test.domain.         IN      A       127.0.0.1
ldap-server.my.test.domain.             IN      A       127.0.0.1
_ldap._tcp.my.test.domain.      IN      SRV     0       0       389
 ldap-server
===== 8< =====

4.  Start up bind:

# service named onestart

5.  Confirm that SRV lookups find our record:

$ dig @localhost _ldap._tcp.my-domain.com SRV
...
;; ANSWER SECTION:
_ldap._tcp.my-domain.com. 10    IN      SRV     0 0 389
ldap-server.my-domain.com.

6.  Tell your system libraries to use this DNS server by temporarily
changing /etc/resolv.conf to say:

===== 8< =====
nameserver 127.0.0.1
===== 8< =====

7.  Confirm that the OpenLDAP tools can look that SRV record up:

$ ldapsearch -H 'ldap:///ou%3Dblah%2Cdc%3Dmy-domain%2Cdc%3Dcom'

(That's "ou=blah,dc=my-domain,dc=com" URL-encoded, from which
"my-domain.com" will be extracted.)  You should see that it's trying
to connect to ldap-server port 389, and you can stick 'x' on the end
of it to see what it looks like when it can't find a SRV record, as a
sanity check:

$ ldapsearch -H 'ldap:///ou%3Dblah%2Cdc%3Dmy-domain%2Cdc%3Dcomx'
DNS SRV: Could not turn domain=my-domain.comx into a hostlist

8.  Set up an LDAP server listening on localhost port 389, and create
a user, such that you can actually authenticate from PostgreSQL with
it.  Gory details omitted.  First test that you can log in with LDAP
authentication when using a pg_hba.conf line like this:

host all fred 127.0.0.1/32 ldap
ldapurl="ldap://ldap-server.my-domain.com/dc=my-domain,dc=com?cn?sub"

9.  Next apply the patch and verify that you can take out the hostname
and let it be discovered via DNS SRV:

host all fred 127.0.0.1/32 ldap ldapurl="ldap:///dc=my-domain,dc=com?cn?sub"

(You can stick some elog(LOG, ...) lines into
InitializeLDAPConnection() if you want to check that
ldap_domain2hostlist() is in fact finding the hostname and port.)

This is a first draft.  Not tested much yet.  I wonder if
HAVE_LDAP_INITIALIZE is a reasonable way to detact OpenLDAP.  The
documentation was written in about 7 seconds so probably needs work.
There is probably a Windowsy way to do this too but I didn't look into
that.

-- 
Thomas Munro
http://www.enterprisedb.com

  2. Re: DNS SRV support for LDAP authentication

    Thomas Munro <thomas.munro@enterprisedb.com> — 2018-09-25T02:20:23Z

    On Tue, Sep 25, 2018 at 2:09 PM Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> 2.  Define a new zone for testing, by adding the following to the end
> 3.  Create that zone file in /usr/local/etc/namedb/master/my.test.domain:

Oops, I changed my testing domain name in the middle of my experiment,
but pasted the older version into the previous message.  Here are the
corrected steps 2 and 3, consistent with the rest:

===== end of /usr/local/etc/namedb/named.conf =====
zone "my-domain.com" {
        type master;
        file "/usr/local/etc/namedb/master/my-domain.com";
};
=====

===== /usr/local/etc/namedb/master/my-domain.com =====
$TTL    10
@       IN      SOA     ns.my-domain.com. admin.my-domain.com. (
                              2         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
        IN      NS      ns.my-domain.com.
ns.my-domain.com.       IN      A       127.0.0.1
my-domain.com.          IN      A       127.0.0.1
ldap-server.my-domain.com.              IN      A       127.0.0.1
_ldap._tcp.my-domain.com.       IN      SRV     0       0       389
 ldap-server
=====

-- 
Thomas Munro
http://www.enterprisedb.com

  3. Re: DNS SRV support for LDAP authentication

    Thomas Munro <thomas.munro@enterprisedb.com> — 2018-11-07T03:39:59Z

    On Tue, Sep 25, 2018 at 2:09 PM Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> Some people like to use DNS SRV records to advertise LDAP servers on
> their network.  Microsoft Active Directory is usually (always?) set up
> that way.  Here is a patch to allow our LDAP auth module to support
> that kind of discovery.  It copies the convention of the OpenLDAP
> command line tools: if you give it a URL that has no hostname, it'll
> try to extract a domain name from the bind DN, and then ask your DNS
> server for a SRV record for LDAP-over-TCP at that domain.  The
> OpenLDAP version of libldap.so exports the magic to do that, so the
> patch is very small (but the infrastructure set-up to test it is a bit
> of a schlep, see below).  I'll add this to the next Commitfest.
>
> [long tedious explanation of how to set up a test with BIND and OpenLDAP on Unix]

Of course the point of this is not really for the Unix-based set-up I
described, but for Microsoft environments with one or more AD servers
and a PostgreSQL server running on (eg) Linux that wants to find AD.
In such environments, from what I can tell, the following should work:

Standard DNS lookup tools should be able to find SRV records
advertising the host, port and weight (priority) of any AD servers on
the network:
$ nslookup -type=any _ldap._tcp.YOUR.DOMAIN
$ dig srv _ldap._tcp.YOUR.DOMAIN
$ host -t srv _ldp._tcp.YOUR.DOMAIN

OpenLDAP command line tools should be able to find the AD server via
those SRV records, extracting YOUR.DOMAIN from the base DN:
$ ldapsearch -H 'ldap:///dc%3DYOUR%2Cdc%3DDOMAIN' ...

pg_hba.conf with an explicit LDAP server name should be able to talk
to Active Directory without using this patch with something like:
host all all 127.0.0.1/32 ldap
ldapurl="ldap://YOUR-AD-SERVER.YOUR.DOMAIN/dc=YOUR,dc=DOMAIN?cn?sub"

pg_hba.conf using this patch should be able to discover the LDAP
server via SRV if you take out the server name:
host all all 127.0.0.1/32 ldap ldapurl="ldap:///dc=YOUR,dc=DOMAIN?cn?sub"

I'm hoping someone can help test this in a real Active Directory environment.

-- 
Thomas Munro
http://www.enterprisedb.com

  4. Re: DNS SRV support for LDAP authentication

    Thomas Munro <thomas.munro@enterprisedb.com> — 2018-11-16T01:52:55Z

    On Wed, Nov 7, 2018 at 4:39 PM Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> On Tue, Sep 25, 2018 at 2:09 PM Thomas Munro
> <thomas.munro@enterprisedb.com> wrote:
> > Some people like to use DNS SRV records to advertise LDAP servers on
> > their network.  Microsoft Active Directory is usually (always?) set up
> > that way.  Here is a patch to allow our LDAP auth module to support
> > that kind of discovery.

Rebased.

I took the liberty of CCing Mark Cave-Ayland, who had some great
advice on the last round of LDAP feature tweaks[1].  Mark, if you have
any comments on the sanity of this proposal, they'd be much
appreciated, otherwise of course please feel free to ignore.  Thanks!

[1] https://www.postgresql.org/message-id/flat/CAEepm%3D0XTkYvMci0WRubZcf_1am8%3DgP%3D7oJErpsUfRYcKF2gwg%40mail.gmail.com

--
Thomas Munro
http://www.enterprisedb.com

  5. Re: DNS SRV support for LDAP authentication

    Daniel Gustafsson <daniel@yesql.se> — 2019-02-01T11:48:15Z

    > On 25 Sep 2018, at 04:09, Thomas Munro <thomas.munro@enterprisedb.com> wrote:

> Some people like to use DNS SRV records to advertise LDAP servers on
> their network.  Microsoft Active Directory is usually (always?) set up
> that way.  Here is a patch to allow our LDAP auth module to support
> that kind of discovery.  It copies the convention of the OpenLDAP
> command line tools: if you give it a URL that has no hostname, it'll
> try to extract a domain name from the bind DN, and then ask your DNS
> server for a SRV record for LDAP-over-TCP at that domain.  The
> OpenLDAP version of libldap.so exports the magic to do that, so the
> patch is very small (but the infrastructure set-up to test it is a bit
> of a schlep, see below).  I'll add this to the next Commitfest.

Sounds like a reasonable feature.

> Testing instructions for (paths and commands given for FreeBSD, adjust
> as appropriate):

Trying this quickly on macOS while at a conference didn’t yield much success,
will do another attempt when I’m on a more reliable connection.

> This is a first draft.  Not tested much yet.  I wonder if
> HAVE_LDAP_INITIALIZE is a reasonable way to detact OpenLDAP.  The
> documentation was written in about 7 seconds so probably needs work.
> There is probably a Windowsy way to do this too but I didn't look into
> that.

Reading through the patch, and related OpenLDAP code, this seems like a good
approach.  A few small comments:

+     If <productname>PostgreSQL</productname> was compiled with OpenLDAP as

Should OpenLDAP be wrapped in <productname> tags as well?  If so, there is
another “bare” instance in client-auth.sgml which perhaps can be wrapped into
this patch while at it.

+               ereport(LOG,
+                       (errmsg("could not look up a hostlist for %s",
+                               domain)));

Should this be \”%s\”?

+               new_uris = psprintf("%s%s%s://%s:%d",

While this construction isn't introduced in this patch, would it not make sense
to convert uris to StringInfo instead to improve readability?

+               /* Step over this hostname and any spaces. */

Nitpicking on a moved hunk, but single-line comments shouldn’t end in a period
I believe.

cheers ./daniel

  6. Re: DNS SRV support for LDAP authentication

    Graham Leggett <minfrin@sharp.fm> — 2019-02-01T22:25:49Z

    On 25 Sep 2018, at 04:09, Thomas Munro <thomas.munro@enterprisedb.com> wrote:

> Some people like to use DNS SRV records to advertise LDAP servers on
> their network.  Microsoft Active Directory is usually (always?) set up
> that way.  Here is a patch to allow our LDAP auth module to support
> that kind of discovery.

Does this support SSL/TLS?

Regards,
Graham
—

  7. Re: DNS SRV support for LDAP authentication

    Thomas Munro <thomas.munro@enterprisedb.com> — 2019-02-01T23:57:27Z

    On Sat, Feb 2, 2019 at 9:25 AM Graham Leggett <minfrin@sharp.fm> wrote:
> On 25 Sep 2018, at 04:09, Thomas Munro <thomas.munro@enterprisedb.com> wrote:
> > Some people like to use DNS SRV records to advertise LDAP servers on
> > their network.  Microsoft Active Directory is usually (always?) set up
> > that way.  Here is a patch to allow our LDAP auth module to support
> > that kind of discovery.
>
> Does this support SSL/TLS?

I didn't try it myself but I found several claims that it works.  I
see complaints that it always looks for _ldap._tcp and not _ldaps._tcp
as you might expect when using ldascheme=ldaps, but that doesn't seem
to be a big problem.  As for ldaptls=1, that must work because it
doesn't even negotiate that until after the connection is made.

-- 
Thomas Munro
http://www.enterprisedb.com

  8. Re: DNS SRV support for LDAP authentication

    Graham Leggett <minfrin@sharp.fm> — 2019-02-02T11:34:56Z

    On 02 Feb 2019, at 01:57, Thomas Munro <thomas.munro@enterprisedb.com> wrote:

> On Sat, Feb 2, 2019 at 9:25 AM Graham Leggett <minfrin@sharp.fm> wrote:
>> On 25 Sep 2018, at 04:09, Thomas Munro <thomas.munro@enterprisedb.com> wrote:
>>> Some people like to use DNS SRV records to advertise LDAP servers on
>>> their network.  Microsoft Active Directory is usually (always?) set up
>>> that way.  Here is a patch to allow our LDAP auth module to support
>>> that kind of discovery.
>> 
>> Does this support SSL/TLS?
> 
> I didn't try it myself but I found several claims that it works.  I
> see complaints that it always looks for _ldap._tcp and not _ldaps._tcp
> as you might expect when using ldascheme=ldaps, but that doesn't seem
> to be a big problem.  As for ldaptls=1, that must work because it
> doesn't even negotiate that until after the connection is made.

If the LDAP server was bound to port 636, how would the client know to use a direct SSL/TLS connection and not STARTTLS?

Regards,
Graham
—

  9. Re: DNS SRV support for LDAP authentication

    Thomas Munro <thomas.munro@enterprisedb.com> — 2019-02-02T12:44:30Z

    On Sat, Feb 2, 2019 at 10:34 PM Graham Leggett <minfrin@sharp.fm> wrote:
> On 02 Feb 2019, at 01:57, Thomas Munro <thomas.munro@enterprisedb.com> wrote:
> > On Sat, Feb 2, 2019 at 9:25 AM Graham Leggett <minfrin@sharp.fm> wrote:
> >> Does this support SSL/TLS?
> > I didn't try it myself but I found several claims that it works.  I
> > see complaints that it always looks for _ldap._tcp and not _ldaps._tcp
> > as you might expect when using ldascheme=ldaps, but that doesn't seem
> > to be a big problem.  As for ldaptls=1, that must work because it
> > doesn't even negotiate that until after the connection is made.
>
> If the LDAP server was bound to port 636, how would the client know to use a direct SSL/TLS connection and not STARTTLS?

SRV records don't control that, so it looks like the person
configuring pg_hba.conf would simply have to know which of the
following formats to use:

ldapurl=ldap:///dc=example,dc=net
ldapurl=ldap:///dc=example,dc=net ldaptls=1
ldapurl=ldaps:///dc=example,dc=net

Only the port and host are obtained from the SRV record, not those
protocol details.  Nothing in RFC 2782 prevents you from setting up
separate "_ldaps._tcp" SRV records (and I can find discussions of that
idea on the net) and then writing custom resolver code that knows to
look for that, but the OpenLDAP code we're using (for compatibility
with the command line tools) is hard coded to use "_ldap._tcp"
always[1].  Active Directory apparently automatically creates only
"_ldap._tcp" SRV records according to its documentation and that's the
user base I was aiming for with this patch, so I think it makes sense
to just use the routines they provide, despite this weakness.

[1] https://github.com/openldap/openldap/blob/b06f5b0493937fc28f2cc86df1d7f464aa4504d8/libraries/libldap/dnssrv.c#L276

-- 
Thomas Munro
http://www.enterprisedb.com

  10. Re: DNS SRV support for LDAP authentication

    Thomas Munro <thomas.munro@enterprisedb.com> — 2019-02-16T09:56:23Z

    On Sat, Feb 2, 2019 at 12:48 AM Daniel Gustafsson <daniel@yesql.se> wrote:
> +               new_uris = psprintf("%s%s%s://%s:%d",
>
> While this construction isn't introduced in this patch, would it not make sense
> to convert uris to StringInfo instead to improve readability?

Yeah.  This coding is ugly and StringInfo would be much nicer.
Thinking about that made me realise that the proposed SRV case should
also handle multiple SRV records by building a multi-URL string too
(instead of just taking the first one).  I will make it so.

-- 
Thomas Munro
http://www.enterprisedb.com

  11. Re: DNS SRV support for LDAP authentication

    Thomas Munro <thomas.munro@gmail.com> — 2019-03-19T08:01:17Z

    On Sat, Feb 16, 2019 at 10:57 PM Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> Yeah.  This coding is ugly and StringInfo would be much nicer.
> Thinking about that made me realise that the proposed SRV case should
> also handle multiple SRV records by building a multi-URL string too
> (instead of just taking the first one).  I will make it so.

Done, in the attached.  Reviewing your comments again, from the top:

On Sat, Feb 2, 2019 at 12:48 AM Daniel Gustafsson <daniel@yesql.se> wrote:
> +     If <productname>PostgreSQL</productname> was compiled with OpenLDAP as
>
> Should OpenLDAP be wrapped in <productname> tags as well?  If so, there is
> another “bare” instance in client-auth.sgml which perhaps can be wrapped into
> this patch while at it.

Fixed.

> +               ereport(LOG,
> +                       (errmsg("could not look up a hostlist for %s",
> +                               domain)));
>
> Should this be \”%s\”?

Yep, fixed.

> +               new_uris = psprintf("%s%s%s://%s:%d",
>
> While this construction isn't introduced in this patch, would it not make sense
> to convert uris to StringInfo instead to improve readability?

Agreed, fixed.

> +               /* Step over this hostname and any spaces. */
>
> Nitpicking on a moved hunk, but single-line comments shouldn’t end in a period
> I believe.

Huh. And yet they are sentences.

tmunro@dogmatix $ git grep '/\* [A-Za-z].*\. \*/' | wc -l
    5607
tmunro@dogmatix $ git grep '/\* [A-Za-z].*[a-z] \*/' | wc -l
   59500

Yep, you win!

I also fixed a bug where some error messages could pass a NULL pointer
for %s when we don't have a server name.

I also added a hint to the error message you get if it can't find DNS
SRV records, so that if you accidentally activate this feature by
forgetting to set the server name, it'll remind you that you could do
that:

 LOG:  LDAP authentication could not find DNS SRV records for "example.net"
 HINT:  Set an LDAP server name explicitly.

Unfortunately, no feedback from MS Active Directory users has been
forthcoming, but I guess that might take a beta release.  See below
for new more complete instructions for testing this with an open
source stack (now that I know there is a lazy way to stand up an LDAP
server using the TAP test stuff, I've adjusted the instructions to
work with that).

I'd like to commit this soon.

Some random things I noticed that I am not fixing in this patch but
wanted to mention:  I don't like the asymmetry initStringInfo(si),
pfree(si->data).  I don't like si->data as a way to get a C string
from a StringInfo.  There are a couple of references to StringBuffer
that surely mean StringInfo in comments.

=== How to test ===

1.  Start up an LDAP server that has a user test1/secret1 under
dc=example,dc=net (it runs in the background and you can stop it with
SIGINT):

$ make -C src/test/ldap check
$ /usr/local/libexec/slapd -f src/test/ldap/tmp_check/slapd.conf -h
ldap://127.0.0.1:5555

2.  Start up a BIND daemon that has multiple SRV records for LDAP at
example.com:

$ tail -4 /usr/local/etc/namedb/named.conf
zone "example.net" {
        type master;
        file "/usr/local/etc/namedb/master/example.net";
};
$ cat /usr/local/etc/namedb/master/example.net
$TTL    10
@       IN      SOA     ns.example.net. admin.example.net. (
                              2         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
        IN      NS      ns.example.net.
ns.example.net. IN A 127.0.0.1
example.net. IN A 127.0.0.1
ldap1.example.net. IN A 127.0.0.1
ldap2.example.net. IN A 127.0.0.1
_ldap._tcp.example.net. IN SRV 0 0 5555 ldap1
_ldap._tcp.example.net. IN SRV 1 0 5555 ldap2

3.  Tell your OS to talk to that DNS server (and, erm, keep what you
had here so you can restore it later):

$ cat /etc/resolv.conf
nameserver 127.0.0.1

4.  Check that standard DNS and LDAP tools can find their way to your
LDAP servers via these breadcrumbs:

$ host -t srv _ldap._tcp.example.net
_ldap._tcp.example.net has SRV record 0 0 5555 ldap1.example.net.
_ldap._tcp.example.net has SRV record 1 0 5555 ldap2.example.net.

$ ldapsearch -H 'ldap:///dc%3Dexample%2Cdc%3Dnet' -b 'dc=example,dc=net'

5.  Tell PostgreSQL to use SRV records in pg_hba.conf using either of
these styles:

host all test1 127.0.0.1/32 ldap basedn="dc=example,dc=net"
host all test1 127.0.0.1/32 ldap ldapurl="ldap:///dc=example,dc=net?uid?sub"

6.  Check that you now log in as test1/secret1:

$ psql -h 127.0.0.1 postgres test1

-- 
Thomas Munro
https://enterprisedb.com

  12. Re: DNS SRV support for LDAP authentication

    Thomas Munro <thomas.munro@gmail.com> — 2019-03-21T03:10:17Z

    On Tue, Mar 19, 2019 at 9:01 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> I'd like to commit this soon.

Done, after some more comment adjustments.  Thanks Daniel and Graham
for your feedback!

-- 
Thomas Munro
https://enterprisedb.com