Horton

Re: Perform COPY FROM encoding conversions in larger chunks

Heikki Linnakangas <hlinnaka@iki.fi>

View thread View on postgr.es
From: Heikki Linnakangas <hlinnaka@iki.fi>
To: John Naylor <john.naylor@enterprisedb.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2021-02-09T17:36:10Z
Lists: pgsql-hackers 
On 09/02/2021 15:40, John Naylor wrote:
> On Sun, Feb 7, 2021 at 2:13 PM Heikki Linnakangas <hlinnaka@iki.fi 
> <mailto:hlinnaka@iki.fi>> wrote:
>  >
>  > On 02/02/2021 23:42, John Naylor wrote:
>  > >
>  > > In copyfromparse.c, this is now out of date:
>  > >
>  > >   * Read the next input line and stash it in line_buf, with 
> conversion to
>  > >   * server encoding.
> 
> This comment for CopyReadLine() is still there. Conversion already 
> happened by now, so I think this comment is outdated.
> 
> Other than that, I think this is ready for commit.

Fixed. And also fixed one more bug in allocating raw_buf_size, the "+ 1" 
somehow went missing again. That was causing a failure on Windows at 
cfbot.cputube.org.

I'll read through this one more time with fresh eyes tomorrow or the day 
after, and push. Thanks for all the review!

- Heikki

Commits

  1. Do COPY FROM encoding conversion/verification in larger chunks.

  2. Add 'noError' argument to encoding conversion functions.

  3. Add direct conversion routines between EUC_TW and Big5.

  4. Add mbverifystr() functions specific to each encoding.