Horton

Thread

Commits

  1. Enforce translation mode for Windows frontends to text with open/fopen

  2. Fix pgbench lexer's "continuation" rule to cope with Windows newlines.

  3. Allow concurrent-safe open() and fopen() in frontend code for Windows

  1. pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Michael Paquier <michael@paquier.xyz> — 2018-09-14T01:05:47Z

    Allow concurrent-safe open() and fopen() in frontend code for Windows

PostgreSQL uses a custom wrapper for open() and fopen() which is
concurrent-safe, allowing multiple processes to open and work on the
same file.  This has a couple of advantages:
- pg_test_fsync does not handle O_DSYNC correctly otherwise, leading to
false claims that disks are unsafe.
- TAP tests can run into race conditions when a postmaster and pg_ctl
open postmaster.pid, fixing some random failures in the buildfam.

pg_upgrade is one frontend tool using workarounds to bypass file locking
issues with the log files it generates, however the interactions with
pg_ctl are proving to be tedious to get rid of, so this is left for
later.

Author: Laurenz Albe
Reviewed-by: Michael Paquier, Kuntal Ghosh
Discussion: https://postgr.es/m/1527846213.2475.31.camel@cybertec.at
Discussion: https://postgr.es/m/16922.1520722108@sss.pgh.pa.us

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/0ba06e0bfb8cfd24ff17aca92aa72245ddd6c4d7

Modified Files
--------------
src/bin/initdb/initdb.c                           | 8 ++++++++
src/bin/pg_basebackup/pg_receivewal.c             | 2 +-
src/bin/pg_verify_checksums/pg_verify_checksums.c | 2 +-
src/common/file_utils.c                           | 4 ++--
src/include/port.h                                | 3 ---
5 files changed, 12 insertions(+), 7 deletions(-)

  2. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-09-14T15:22:55Z

    Michael Paquier <michael@paquier.xyz> writes:
> Allow concurrent-safe open() and fopen() in frontend code for Windows

I'm surprised you didn't back-patch this --- isn't it a bug fix?

A compromise might be to push it to v11 but not further.

			regards, tom lane

  3. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Michael Paquier <michael@paquier.xyz> — 2018-09-14T22:49:39Z

    On Fri, Sep 14, 2018 at 11:22:55AM -0400, Tom Lane wrote:
> I'm surprised you didn't back-patch this --- isn't it a bug fix?
> 
> A compromise might be to push it to v11 but not further.

Yes, that's clearly a bug fix from pg_ctl point of view with TAP tests.
However, I have rather cold feet about back-patching for two reasons as
this introduces a behavior change:
- The concurrency part disappears.
- Caller of open() needs to enforce text mode to strip correctly CRLF
characters.

I would be fine to get that into v11 though, as that is not released
yet.  You will have to wait for a couple of days though, there is a long
week-end here away from laptops and electronic devices ;)
--
Michael

  4. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Michael Paquier <michael@paquier.xyz> — 2018-09-17T11:41:13Z

    On Sat, Sep 15, 2018 at 07:49:39AM +0900, Michael Paquier wrote:
> I would be fine to get that into v11 though, as that is not released
> yet.  You will have to wait for a couple of days though, there is a long
> week-end here away from laptops and electronic devices ;)

OK, REL_11_STABLE has been patched as well, after doing a couple of
extra tests on Windows.
--
Michael

  5. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-09-17T13:41:37Z

    Michael Paquier <michael@paquier.xyz> writes:
> OK, REL_11_STABLE has been patched as well, after doing a couple of
> extra tests on Windows.

BTW, I'm a bit concerned by the fact that bowerbird has failed its
last couple of HEAD runs at the pgbench step.  The first such
failure was here:

https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=bowerbird&dt=2018-09-15%2014%3A19%3A58

Looking at the set of commits between the prior run and that one,
it's hard to see anything that could have triggered the test failures
other than this patch --- but I also don't see how this patch would've
blown up pgbench without breaking earlier tests.  Ideas?

			regards, tom lane

  6. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Michael Paquier <michael@paquier.xyz> — 2018-09-17T14:02:02Z

    On Mon, Sep 17, 2018 at 09:41:37AM -0400, Tom Lane wrote:
> BTW, I'm a bit concerned by the fact that bowerbird has failed its
> last couple of HEAD runs at the pgbench step.  The first such
> failure was here:
> 
> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=bowerbird&dt=2018-09-15%2014%3A19%3A58
> 
> Looking at the set of commits between the prior run and that one,
> it's hard to see anything that could have triggered the test failures
> other than this patch --- but I also don't see how this patch would've
> blown up pgbench without breaking earlier tests.  Ideas?

Thanks, I have been looking at the build farm but I missed this one.
dory, which uses VS 2015 is not complaining because it does not run
bincheck.  At quick glance, it seems to be caused by process_file() in
pgbench.c which would need to open files in text mode, and the input
file parsing fails at the first '\' character found.  I'll test that
stuff on tomorrow morning manually.
--
Michael

  7. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-09-17T14:48:32Z

    Michael Paquier <michael@paquier.xyz> writes:
> On Mon, Sep 17, 2018 at 09:41:37AM -0400, Tom Lane wrote:
>> Looking at the set of commits between the prior run and that one,
>> it's hard to see anything that could have triggered the test failures
>> other than this patch --- but I also don't see how this patch would've
>> blown up pgbench without breaking earlier tests.  Ideas?

> Thanks, I have been looking at the build farm but I missed this one.
> dory, which uses VS 2015 is not complaining because it does not run
> bincheck.  At quick glance, it seems to be caused by process_file() in
> pgbench.c which would need to open files in text mode, and the input
> file parsing fails at the first '\' character found.

Oh, you're thinking pgbench isn't robust against finding \r's visible
in its input?  Could be.

> I'll test that stuff on tomorrow morning manually.

We've got a bit of a timing problem because we want to wrap 11beta4/rc1
(still TBD) in a few hours.  I'll take a look and see if I can push a
quick fix before that.

			regards, tom lane

  8. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> — 2018-09-17T15:13:28Z

    
On 09/17/2018 10:48 AM, Tom Lane wrote:
> Michael Paquier <michael@paquier.xyz> writes:
>> On Mon, Sep 17, 2018 at 09:41:37AM -0400, Tom Lane wrote:
>>> Looking at the set of commits between the prior run and that one,
>>> it's hard to see anything that could have triggered the test failures
>>> other than this patch --- but I also don't see how this patch would've
>>> blown up pgbench without breaking earlier tests.  Ideas?
>> Thanks, I have been looking at the build farm but I missed this one.
>> dory, which uses VS 2015 is not complaining because it does not run
>> bincheck.  At quick glance, it seems to be caused by process_file() in
>> pgbench.c which would need to open files in text mode, and the input
>> file parsing fails at the first '\' character found.
> Oh, you're thinking pgbench isn't robust against finding \r's visible
> in its input?  Could be.
>
>> I'll test that stuff on tomorrow morning manually.
> We've got a bit of a timing problem because we want to wrap 11beta4/rc1
> (still TBD) in a few hours.  I'll take a look and see if I can push a
> quick fix before that.


When you do I'll start a bowerbird run to check it.

cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  9. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-09-17T16:13:56Z

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> On 09/17/2018 10:48 AM, Tom Lane wrote:
>> We've got a bit of a timing problem because we want to wrap 11beta4/rc1
>> (still TBD) in a few hours.  I'll take a look and see if I can push a
>> quick fix before that.

> When you do I'll start a bowerbird run to check it.

Pushed, please test.

I think there's a general issue here of exactly how we want pgwin32_fopen
to behave, but the immediate problem is best fixed by making pgbench deal
with Windows newlines more thoroughly.

			regards, tom lane

  10. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> — 2018-09-17T18:30:40Z

    
On 09/17/2018 12:13 PM, Tom Lane wrote:
> Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
>> On 09/17/2018 10:48 AM, Tom Lane wrote:
>>> We've got a bit of a timing problem because we want to wrap 11beta4/rc1
>>> (still TBD) in a few hours.  I'll take a look and see if I can push a
>>> quick fix before that.
>> When you do I'll start a bowerbird run to check it.
> Pushed, please test.
>
> I think there's a general issue here of exactly how we want pgwin32_fopen
> to behave, but the immediate problem is best fixed by making pgbench deal
> with Windows newlines more thoroughly.
>
> 			


Tests are still running, but it's past the pgbench stage on HEAD, so I 
think we're good.

cheers

andrew


-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  11. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-09-17T22:00:58Z

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> On 09/17/2018 12:13 PM, Tom Lane wrote:
>> Pushed, please test.

> Tests are still running, but it's past the pgbench stage on HEAD, so I 
> think we're good.

I was more concerned about whether any of the post-pgbench steps would
show a failure; but it looks like HEAD's green, so probably v11 is too.

			regards, tom lane

  12. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-09-17T23:38:24Z

    So we seem to be out of the woods in terms of 0ba06e0bf breaking the
regression tests, but I'm not very happy about the whole thing, because
that patch wasn't supposed to change the behavior of open/fopen in any
way other than allowing concurrent file access.  Obviously, it did.

After looking at src/port/open.c a bit, it seems like the problem is
that our fopen() requires a nonstandard "t" option to select text mode.
I'd always supposed that you got binary mode with "b" and text mode
otherwise; is there some third possibility on Windows?

Anyway, I'm inclined to propose that we ought to do something like
the attached in HEAD and v11 in order to reduce the risk that there
are other unexpected behavioral changes.  This should also let us
revert 0ba06e0bf's change in initdb.c.

I wonder whether pgwin32_open() also ought to enforce that either
O_BINARY or O_TEXT mode gets selected.

			regards, tom lane

  13. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Michael Paquier <michael@paquier.xyz> — 2018-09-18T00:11:43Z

    On Mon, Sep 17, 2018 at 07:38:24PM -0400, Tom Lane wrote:
> So we seem to be out of the woods in terms of 0ba06e0bf breaking the
> regression tests, but I'm not very happy about the whole thing, because
> that patch wasn't supposed to change the behavior of open/fopen in any
> way other than allowing concurrent file access.  Obviously, it did.

Thanks!  I can see that as well.

> After looking at src/port/open.c a bit, it seems like the problem is
> that our fopen() requires a nonstandard "t" option to select text mode.
> I'd always supposed that you got binary mode with "b" and text mode
> otherwise; is there some third possibility on Windows?

There is a sort of automatic mode...  Please see below.

> Anyway, I'm inclined to propose that we ought to do something like
> the attached in HEAD and v11 in order to reduce the risk that there
> are other unexpected behavioral changes.  This should also let us
> revert 0ba06e0bf's change in initdb.c.
> 
> I wonder whether pgwin32_open() also ought to enforce that either
> O_BINARY or O_TEXT mode gets selected.

There is no need to go down that road I think, a lot of code paths
already call setmode to make sure that binary or text modes are used.

> diff --git a/src/port/open.c b/src/port/open.c
> index a3ad946..85ab06e 100644
> --- a/src/port/open.c
> +++ b/src/port/open.c
> @@ -154,7 +154,7 @@ pgwin32_fopen(const char *fileName, const char *mode)
>  
>  	if (strchr(mode, 'b'))
>  		openmode |= O_BINARY;
> -	if (strchr(mode, 't'))
> +	else
>  		openmode |= O_TEXT;

Hm.  I don't think that this is correct either.  The whole logic behind
the mode selected depends on how setmode() is being set to.  Please see
here:
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/setmode?view=vs-2017

And the point is that if neither 'b' nor 't' are set, then open() would
use the value set by the process, which is 't' by default if not set.
And initdb does not set that, so it would use 't'.  miscinit.c sets it
to 'b', and your patch would likely cause some backend code to be
broken.  So it seems to me that if 'b' or 't' are not defined by the
caller, then we ought to use what get_fmode() returns:
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/get-fmode?view=vs-2017

What I think I broke is that CreateFile ignores what _fmode uses, which
has caused the breakage, while calling directly open() or fopen() does
the work.  There are also other things assuming that binary mode is
used, you can grep for "setmode" and see how miscinit.c or pg_basebackup
do the job.
--
Michael

  14. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Michael Paquier <michael@paquier.xyz> — 2018-09-18T02:42:27Z

    On Tue, Sep 18, 2018 at 09:11:43AM +0900, Michael Paquier wrote:
> What I think I broke is that CreateFile ignores what _fmode uses, which
> has caused the breakage, while calling directly open() or fopen() does
> the work.  There are also other things assuming that binary mode is
> used, you can grep for "setmode" and see how miscinit.c or pg_basebackup
> do the job.

I have been playing with this stuff, and hacked the attached.  Now, while
TAP tests of initdb and pgbench are happy (I can actually see the past
failure as well), pg_dump complains at authentication time when using
plain-text mode when using databases with all ASCII characters.  That's
not something I expected first, but _get_fmode also influences
operations like pipe(), which is something that pg_dump uses, and
setmode is enforced to binary mode only when adapted.

I am getting to wonder if what's present on HEAD represents actually the
best deal we could get.  Attached is the patch I used for reference.
Thoughts?
--
Michael

  15. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-09-18T14:45:09Z

    Michael Paquier <michael@paquier.xyz> writes:
> On Tue, Sep 18, 2018 at 09:11:43AM +0900, Michael Paquier wrote:
>> What I think I broke is that CreateFile ignores what _fmode uses, which
>> has caused the breakage, while calling directly open() or fopen() does
>> the work.  There are also other things assuming that binary mode is
>> used, you can grep for "setmode" and see how miscinit.c or pg_basebackup
>> do the job.

> I have been playing with this stuff, and hacked the attached.  Now, while
> TAP tests of initdb and pgbench are happy (I can actually see the past
> failure as well), pg_dump complains at authentication time when using
> plain-text mode when using databases with all ASCII characters.  That's
> not something I expected first, but _get_fmode also influences
> operations like pipe(), which is something that pg_dump uses, and
> setmode is enforced to binary mode only when adapted.

> I am getting to wonder if what's present on HEAD represents actually the
> best deal we could get.  Attached is the patch I used for reference.
> Thoughts?

Well, we have to do something.  I have a report from EDB's packagers
that in 11beta4, "initdb --pwfile" is failing on Windows (ie, one can't
connect afterwards using the specified password).  It seems nearly
certain to me that the reason is that the file is read with

		FILE	   *pwf = fopen(pwfilename, "r");

and so the \r isn't getting stripped from what's used as the password.

So my opinion is that it's not negotiable that we have to restore
the previous behavior in this realm.  I don't want to be running
around finding other cases one at a time, and I absolutely won't
hold still for putting an "#ifdef WIN32" around each such case.
(Even if we fixed all our own code, we'd still be breaking 3rd-party
code.)

What I don't understand yet is why your latest patch doesn't restore
the previous behavior.  What exactly is still different?

In the meantime, we might be well advised to revert this patch in
v11 and just continue to work on the problem in HEAD.  I see now
that this wasn't something to cram in during late beta ...

			regards, tom lane

  16. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Laurenz Albe <laurenz.albe@cybertec.at> — 2018-09-18T15:28:53Z

    Tom Lane wrote:
> Well, we have to do something.  I have a report from EDB's packagers
> that in 11beta4, "initdb --pwfile" is failing on Windows (ie, one can't
> connect afterwards using the specified password).  It seems nearly
> certain to me that the reason is that the file is read with
> 
>                 FILE       *pwf = fopen(pwfilename, "r");
> 
> and so the \r isn't getting stripped from what's used as the password.

Perhaps there is something obvious that I'm missing, but it seems that
all the problems we observe are caused by frontend code suddenly defaulting
to binary mode when it was text mode before.

Would it be an option to have pgwin32_open default to text mode in
frontend code and to binary mode in backend code?

Yours,
Laurenz Albe

  17. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-09-18T15:38:57Z

    Laurenz Albe <laurenz.albe@cybertec.at> writes:
> Would it be an option to have pgwin32_open default to text mode in
> frontend code and to binary mode in backend code?

Well, the question is why Michael's latest proposed patch doesn't
accomplish that.

			regards, tom lane

  18. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Laurenz Albe <laurenz.albe@cybertec.at> — 2018-09-18T16:04:58Z

    Tom Lane wrote:
> Laurenz Albe <laurenz.albe@cybertec.at> writes:
> > Would it be an option to have pgwin32_open default to text mode in
> > frontend code and to binary mode in backend code?
> 
> Well, the question is why Michael's latest proposed patch doesn't
> accomplish that.

I was thinking of something trivial like this:

--- a/src/port/open.c
+++ b/src/port/open.c
@@ -71,6 +71,12 @@ pgwin32_open(const char *fileName, int fileFlags,...)
                         _O_SHORT_LIVED | O_DSYNC | O_DIRECT |
                         (O_CREAT | O_TRUNC | O_EXCL) | (O_TEXT | O_BINARY))) == fileFlags);
 
+#ifdef FRONTEND
+   /* default to text mode in frontend code */
+   if (fileFlags & O_BINARY == 0)
+       fileFlags |= O_TEXT;
+#endif
+
    sa.nLength = sizeof(sa);
    sa.bInheritHandle = TRUE;
    sa.lpSecurityDescriptor = NULL;

That wouldn't influence pipes, which was what Michael said was a
problem for pg_dump.

I currently have no Windows system close, so I cannot test...

Yours,
Laurenz Albe

  19. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Michael Paquier <michael@paquier.xyz> — 2018-09-18T21:38:07Z

    On Tue, Sep 18, 2018 at 10:45:09AM -0400, Tom Lane wrote:
> In the meantime, we might be well advised to revert this patch in
> v11 and just continue to work on the problem in HEAD.  I see now
> that this wasn't something to cram in during late beta ...

I can see that you have reverted the change just before I was able to
reply.  Thanks I was going to do the same.  Also, if we cannot find a
clear solution for HEAD rather soon, say by tomorrow, let's also remove
it there as well and go ack to the blackboard.  I still want to test a
couple of other solutions first.
--
Michael

  20. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Michael Paquier <michael@paquier.xyz> — 2018-09-19T02:32:11Z

    On Tue, Sep 18, 2018 at 06:04:58PM +0200, Laurenz Albe wrote:
> That wouldn't influence pipes, which was what Michael said was a
> problem for pg_dump.

Yeah, the authentication blows up badly on that..  You can see all the
tests using user and database names with all ASCII range turning red.

> I currently have no Windows system close, so I cannot test...

I have spent a large portion of my morning trying to test all the
solutions proposed, and a winner shows up.  Attached are three patches
which present all the solutions mentioned, attached here for visibility:
- win32-open-michael.patch, or the solution I was working on.  This has
led me actually nowhere.  Even trying to enforce _fmode to happen only
on the frontend has caused breakages of pg_dump for authentication.
Trying to be smarter than what other binaries do is nice and consistent
with the Windows experience I think, still it looks that this can lead
to breakages when a utility uses setmode() for some of the output
files.  I noticed that pgwin32_open missed to enforce the mode used when
calling _fdmode, still that solves nothing.
- win32-open-tom.patch, which switches pgwin32_fopen() to use text mode
all the time if binary is not specified.  While this looked promising,
I have noticed that this has been causing the same set of errors as what
my previous patch has been doing in pg_dump TAP tests.  Anyway, a
solution needs also to happen for pgwin32_open() as we want direct calls
to it to get the right call.
- win32-open-laurenz.patch, which enforces to text mode only if binary
mode is not defined, which maps strictly to what pre-11 is doing when
calling the system _open or _fopen.  And surprisingly, this is proving
to pass all the tests I ran: bincheck (including pgbench and pg_dump),
upgradecheck, recoverycheck, check, etc.  initdb --pwfile is not
complaining to me either.

So the odds are that Laurenz's solution is what we are looking for.
Laurenz, Tom, any thoughts to share?  I won't back-patch that ;)
--
Michael

  21. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-09-19T15:55:01Z

    Michael Paquier <michael@paquier.xyz> writes:
> I have spent a large portion of my morning trying to test all the
> solutions proposed, and a winner shows up.  ...

> - win32-open-laurenz.patch, which enforces to text mode only if binary
> mode is not defined, which maps strictly to what pre-11 is doing when
> calling the system _open or _fopen.  And surprisingly, this is proving
> to pass all the tests I ran: bincheck (including pgbench and pg_dump),
> upgradecheck, recoverycheck, check, etc.  initdb --pwfile is not
> complaining to me either.

I'm OK with this approach.  I wonder though what happens if you take
away the "#ifdef FRONTEND" and just enforce that one or the other mode
is selected always.  That would seem like a sensible solution rather
than a wart to me ...

			regards, tom lane

  22. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Michael Paquier <michael@paquier.xyz> — 2018-09-19T23:58:20Z

    On Wed, Sep 19, 2018 at 11:55:01AM -0400, Tom Lane wrote:
> I'm OK with this approach.  I wonder though what happens if you take
> away the "#ifdef FRONTEND" and just enforce that one or the other mode
> is selected always.  That would seem like a sensible solution rather
> than a wart to me ...

Thanks, I have pushed the solution from Laurenz to maintain pure
compatibility.  The origin of the failures reported by pg_dump actually
comes from a mismatch with pg_hba.conf in the way users and/or databases
are parsed.  I am glad that we have tests for the full range of ASCII
characters in TAP so as it is easy to detect regressions at early
stages.  We could try to manipulate more setmode calls like the one in
miscinit.c so as authentication gets the right call.  I am not sure of
other side effects though...
--
Michael

  23. Re: pgsql: Allow concurrent-safe open() and fopen() in frontend code for Wi

    Laurenz Albe <laurenz.albe@cybertec.at> — 2018-09-20T03:51:20Z

    Michael Paquier wrote:
> Thanks, I have pushed the solution from Laurenz to maintain pure
> compatibility.

Thanks for all the work!

Yours,
Laurenz Albe