Horton

Re: Supporting SJIS as a database encoding

Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com>

View thread View on postgr.es
From: "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com>
To: 'Kyotaro HORIGUCHI' <horiguchi.kyotaro@lab.ntt.co.jp>
Cc: "hlinnaka@iki.fi" <hlinnaka@iki.fi>, "tgl@sss.pgh.pa.us" <tgl@sss.pgh.pa.us>, "ishii@sraoss.co.jp" <ishii@sraoss.co.jp>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Date: 2016-09-08T07:09:51Z
Lists: pgsql-hackers 
From: pgsql-hackers-owner@postgresql.org
> [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Kyotaro
> HORIGUCHI
> <Using radix tree>
> $ time psql postgres -c 'select t.a from t, generate_series(0, 9999)' >
> /dev/null
> 
> real	0m22.696s
> user	0m16.991s
> sys	0m0.182s>
> 
> Using binsearch the result for the same operation was
> real	0m35.296s
> user	0m17.166s
> sys	0m0.216s
> 
> Returning in UTF-8 bloats the result string by about 1.5 times so it doesn't
> seem to make sense comparing with it. But it takes real = 47.35s.

Cool, 36% speedup!  Does this difference vary depending on the actual characters used, e.g. the speedup would be greater if most of the characters are ASCII?

Regards
Takayuki Tsunakawa

Commits

  1. Use radix tree for character encoding conversions.

  2. Small fixes to the Perl scripts to create unicode conversion tables.

  3. Rewrite the perl scripts to produce our Unicode conversion tables.

  4. Remove leading zeros, for consistency with other map files.

  5. Remove code points < 0x80 from character conversion tables.