Re: Perform COPY FROM encoding conversions in larger chunks

Heikki Linnakangas <hlinnaka@iki.fi>

From: Heikki Linnakangas <hlinnaka@iki.fi>
To: John Naylor <john.naylor@enterprisedb.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Date: 2021-01-28T16:43:58Z
Lists: pgsql-hackers

Attachments

On 28/01/2021 15:05, Heikki Linnakangas wrote:
> Next I'm going to write the pg_upgrade check for
> patch 0004, to get that into a committable state too.

As promised, here are new versions of the remaining patches, with the 
pg_upgrade check added. If you have any custom encoding conversions in 
the old cluster, pg_upgrade now fails:

> Performing Consistency Checks
> -----------------------------
> Checking cluster versions                                   ok
> Checking database user is the install user                  ok
> Checking database connection settings                       ok
> Checking for prepared transactions                          ok
> Checking for reg* data types in user tables                 ok
> Checking for contrib/isn with bigint-passing mismatch       ok
> Checking for user-defined encoding conversions              fatal
> 
> Your installation contains user-defined encoding conversions.
> The conversion function parameters changed in PostgreSQL version 14
> so this cluster cannot currently be upgraded.  You can remove the
> encoding conversions in the old cluster and restart the upgrade.
> A list of user-defined encoding conversions is in the file:
>     encoding_conversions.txt
> 
> Failure, exiting

To test this, I wrote a dummy conversion function, also attached.

- Heikki

Commits

  1. Do COPY FROM encoding conversion/verification in larger chunks.

  2. Add 'noError' argument to encoding conversion functions.

  3. Add direct conversion routines between EUC_TW and Big5.

  4. Add mbverifystr() functions specific to each encoding.