Horton

Re: making the backend's json parser work in frontend code

David Steele <david@pgmasters.net>

View thread View on postgr.es
From: David Steele <david@pgmasters.net>
To: Tom Lane <tgl@sss.pgh.pa.us>, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
Cc: Robert Haas <robertmhaas@gmail.com>, Mark Dilger <mark.dilger@enterprisedb.com>, Alvaro Herrera <alvherre@2ndquadrant.com>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2020-01-24T16:36:58Z
Lists: pgsql-hackers 
On 1/24/20 9:27 AM, Tom Lane wrote:
> Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
>> On 2020-01-23 18:04, Robert Haas wrote:
>>> Now, you might say "well, why don't we just do an encoding
>>> conversion?", but we can't. When the filesystem tells us what the file
>>> names are, it does not tell us what encoding the person who created
>>> those files had in mind. We don't know that they had*any*  encoding in
>>> mind. IIUC, a file in the data directory can have a name that consists
>>> of any sequence of bytes whatsoever, so long as it doesn't contain
>>> prohibited characters like a path separator or \0 byte. But only some
>>> of those possible octet sequences can be stored in a manifest that has
>>> to be valid UTF-8.
> 
>> I think it wouldn't be unreasonable to require that file names in the
>> database directory be consistently encoded (as defined by pg_control,
>> probably).  After all, this information is sometimes also shown in
>> system views, so it's already difficult to process total junk.  In
>> practice, this shouldn't be an onerous requirement.
> 
> I don't entirely follow why we're discussing this at all, if the
> requirement is backing up a PG data directory.  There are not, and
> are never likely to be, any legitimate files with non-ASCII names
> in that context.  Why can't we just skip any such files?

It's not uncommon in my experience for users to drop odd files into 
PGDATA (usually versioned copies of postgresql.conf, etc.), but I agree 
that it should be discouraged.  Even so, I don't recall ever seeing any 
non-ASCII filenames.

Skipping files sounds scary, I'd prefer an error or a warning (and then 
base64 encode the filename).

Regards,
-- 
-David
david@pgmasters.net

Commits

  1. Add jsonapi.c to Mkvcbuild.pm's @pgcommonallfiles.

  2. Move jsonapi.c and jsonapi.h to src/common.

  3. Move some code from jsonapi.c to jsonfuncs.c.

  4. Adjust pg_parse_json() so that it does not directly ereport().

  5. Remove jsonapi.c's lex_accept().

  6. Split JSON lexer/parser from 'json' data type support.

  7. Rationalize code placement between wchar.c, encnames.c, and mbutils.c.

  8. Move wchar.c and encnames.c to src/common/.

  9. Update header comments for wchar.c and encnames.c.

  10. Make StringInfo available to frontend code.

  11. Use SASLprep to normalize passwords for SCRAM authentication.