Thread

UTF8 or Unicode

Bruce Momjian <pgman@candle.pha.pa.us> — 2005-02-15T02:14:54Z

Should our multi-byte encoding be referred to as UTF8 or Unicode?
I know UTF8 is a type of unicode but do we need to rename anything from
Unicode to UTF8?

Someone asked me via private email.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: UTF8 or Unicode

Abhijit Menon-Sen <ams@oryx.com> — 2005-02-15T02:27:32Z

At 2005-02-14 21:14:54 -0500, pgman@candle.pha.pa.us wrote:
>
> Should our multi-byte encoding be referred to as UTF8 or Unicode?

The *encoding* should certainly be referred to as UTF-8. Unicode is a
character set, not an encoding; Unicode characters may be encoded with
UTF-8, among other things.

(One might think of a charset as being a set of integers representing
characters, and an encoding as specifying how those integers may be
converted to bytes.)

> I know UTF8 is a type of unicode but do we need to rename anything
> from Unicode to UTF8?

I don't know. I'll go through the documentation to see if I can find
anything that needs changing.

-- ams

Re: UTF8 or Unicode

Bruce Momjian <pgman@candle.pha.pa.us> — 2005-02-15T03:05:08Z

Abhijit Menon-Sen wrote:
> At 2005-02-14 21:14:54 -0500, pgman@candle.pha.pa.us wrote:
> >
> > Should our multi-byte encoding be referred to as UTF8 or Unicode?
> 
> The *encoding* should certainly be referred to as UTF-8. Unicode is a
> character set, not an encoding; Unicode characters may be encoded with
> UTF-8, among other things.
> 
> (One might think of a charset as being a set of integers representing
> characters, and an encoding as specifying how those integers may be
> converted to bytes.)
> 
> > I know UTF8 is a type of unicode but do we need to rename anything
> > from Unicode to UTF8?
> 
> I don't know. I'll go through the documentation to see if I can find
> anything that needs changing.

I looked at encoding.sgml and that mentions Unicode, and then UTF8 as an
acronym. I am wondering if we need to make UTF8 first and Unicode
second.  Does initdb accept UTF8 as an encoding?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: UTF8 or Unicode

A.M. <agentm@themactionfaction.com> — 2005-02-15T03:13:32Z

On Feb 14, 2005, at 9:27 PM, Abhijit Menon-Sen wrote:
>
>> I know UTF8 is a type of unicode but do we need to rename anything
>> from Unicode to UTF8?
>
> I don't know. I'll go through the documentation to see if I can find
> anything that needs changing.
>
It's not the documentation that is wrong. Specifying the database 
"encoding" as "Unicode" is simply a bug (see initdb). What if 
postgresql supports UTF-16 in the future? What would you call it?

Also, the backend protocol also uses "UNICODE" when specifying the 
encoding. All the other encoding names are specified correctly AFAICS.

I brought this up before:
http://archives.postgresql.org/pgsql-hackers/2004-10/msg00811.php

We could make UTF8 the canonical form in the aliasing mechanism, but
beta 4 is a bit late to come up with this kind of idea.
-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

Re: UTF8 or Unicode

Karel Zak <zakkr@zf.jcu.cz> — 2005-02-15T09:22:03Z

On Mon, 2005-02-14 at 22:05 -0500, Bruce Momjian wrote:
> Abhijit Menon-Sen wrote:
> > At 2005-02-14 21:14:54 -0500, pgman@candle.pha.pa.us wrote:
> > >
> > > Should our multi-byte encoding be referred to as UTF8 or Unicode?
> > 
> > The *encoding* should certainly be referred to as UTF-8. Unicode is a
> > character set, not an encoding; Unicode characters may be encoded with
> > UTF-8, among other things.
> > 
> > (One might think of a charset as being a set of integers representing
> > characters, and an encoding as specifying how those integers may be
> > converted to bytes.)
> > 
> > > I know UTF8 is a type of unicode but do we need to rename anything
> > > from Unicode to UTF8?
> > 
> > I don't know. I'll go through the documentation to see if I can find
> > anything that needs changing.
> 
> I looked at encoding.sgml and that mentions Unicode, and then UTF8 as an
> acronym. I am wondering if we need to make UTF8 first and Unicode
> second.  Does initdb accept UTF8 as an encoding?

in PG: unicode = utf8 = utf-8 

Our internal routines in src/backend/utils/mb/encnames.c accept all
synonyms. The "official" internal PG name for UTF-8 is "UNICODE" :-(

It's historical reason that UTF8 = UNICODE, because there was "UNICODE"
first. It's same like "WIN" for WIN1251 (in sources it's marked as
"_dirty_ alias")...

I think initdb uses pg_char_to_encoding() from
src/backend/utils/mb/encnames.c and it should be accept all aliases.

	Karel

-- 
Karel Zak <zakkr@zf.jcu.cz>

Re: UTF8 or Unicode

Peter Eisentraut <peter_e@gmx.net> — 2005-02-15T13:33:29Z

Am Dienstag, 15. Februar 2005 10:22 schrieb Karel Zak:
> in PG: unicode = utf8 = utf-8
>
> Our internal routines in src/backend/utils/mb/encnames.c accept all
> synonyms. The "official" internal PG name for UTF-8 is "UNICODE" :-(

I think in the SQL standard the official name is UTF8.  If someone wants to 
verify that this is the case and is exactly the encoding we offer (perhaps 
modulo the 0x10000 issue), then it might make sense to change the canonical 
form to UTF8.

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

Re: UTF8 or Unicode

Karel Zak <zakkr@zf.jcu.cz> — 2005-02-18T09:28:00Z

On Tue, 2005-02-15 at 14:33 +0100, Peter Eisentraut wrote:
> Am Dienstag, 15. Februar 2005 10:22 schrieb Karel Zak:
> > in PG: unicode = utf8 = utf-8
> >
> > Our internal routines in src/backend/utils/mb/encnames.c accept all
> > synonyms. The "official" internal PG name for UTF-8 is "UNICODE" :-(
> 
> I think in the SQL standard the official name is UTF8.  If someone wants to 
> verify that this is the case and is exactly the encoding we offer (perhaps 
> modulo the 0x10000 issue), then it might make sense to change the canonical 
> form to UTF8.

Yes, I think we should fix it and remove UNICODE and WIN encoding names
from PG code.

	Karel

-- 
Karel Zak <zakkr@zf.jcu.cz>

Re: UTF8 or Unicode

Oliver Jowett <oliver@opencloud.com> — 2005-02-18T11:27:06Z

Karel Zak wrote:

> Yes, I think we should fix it and remove UNICODE and WIN encoding names
> from PG code.

The JDBC driver asks for a UNICODE client encoding before it knows the 
server version it is talking to. How do you avoid breaking this?

-O

Re: UTF8 or Unicode

Karel Zak <zakkr@zf.jcu.cz> — 2005-02-18T11:53:02Z

On Sat, 2005-02-19 at 00:27 +1300, Oliver Jowett wrote:
> Karel Zak wrote:
> 
> > Yes, I think we should fix it and remove UNICODE and WIN encoding names
> > from PG code.
> 
> The JDBC driver asks for a UNICODE client encoding before it knows the 
> server version it is talking to. How do you avoid breaking this?

Fix JDBC driver as soon as possible.

Add to 8.1 release notes: encoding names 'UNICODE' and 'WIN' are
deprecated and it will removed in next release. Please, use correct
names "UTF-8" and "WIN1215".

8.2: remove it.

OK?

	Karel

-- 
Karel Zak <zakkr@zf.jcu.cz>

Re: UTF8 or Unicode

Christopher Kings-Lynne <chriskl@familyhealth.com.au> — 2005-02-18T11:56:32Z

> Add to 8.1 release notes: encoding names 'UNICODE' and 'WIN' are
> deprecated and it will removed in next release. Please, use correct
> names "UTF-8" and "WIN1215".
> 
> 8.2: remove it.
> 
> OK?

Why on earth remove it?  Just leave it in as an alias to UTF8

Chris

Re: UTF8 or Unicode

Oliver Jowett <oliver@opencloud.com> — 2005-02-18T12:14:35Z

Karel Zak wrote:
> On Sat, 2005-02-19 at 00:27 +1300, Oliver Jowett wrote:
> 
>>Karel Zak wrote:
>>
>>
>>>Yes, I think we should fix it and remove UNICODE and WIN encoding names
>>>from PG code.
>>
>>The JDBC driver asks for a UNICODE client encoding before it knows the 
>>server version it is talking to. How do you avoid breaking this?
> 
> Fix JDBC driver as soon as possible.

How, exactly? Ask for a 'utf8' client encoding instead of 'UNICODE'? 
Will this work if the driver is connecting to an older server?

> Add to 8.1 release notes: encoding names 'UNICODE' and 'WIN' are
> deprecated and it will removed in next release. Please, use correct
> names "UTF-8" and "WIN1215".

8.0 appears to spell it 'utf8'.

Removing the existing aliases seems like a fairly gratuitous 
incompatibility to introduce to me.

-O