Horton

Re: Support UTF-8 files with BOM in COPY FROM

Peter Eisentraut <peter_e@gmx.net>

View thread
From: Peter Eisentraut <peter_e@gmx.net>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Tatsuo Ishii <ishii@postgresql.org>, david@kineticode.com, itagaki.takahiro@gmail.com, pgsql-hackers@postgresql.org
Date: 2011-09-26T18:38:32Z
Lists: pgsql-hackers 
On mån, 2011-09-26 at 13:19 -0400, Robert Haas wrote:
> The thing that makes me doubt that is this comment from Tatsuo Ishii:
> 
> TI> COPY explicitly specifies the encoding (to be UTF-8 in this case).
> So
> TI> I think we should not regard U+FEFF as "BOM" in COPY, rather we
> should
> TI> regard U+FEFF as "ZERO WIDTH NO-BREAK SPACE".
> 
> If a BOM is confusable with valid data, then I think recognizing it
> and discarding it unconditionally is no good - you could end up where
> COPY OUT, TRUNCATE, COPY IN changes the table contents.

We did recently accept a patch for psql -f to skip over a UTF-8
byte-order mark.  We had a lot of this same discussion there.