Thread

1. Re: Bulkloading using COPY - ignore duplicates?

   Zeugswetter Andreas ADI SD <zeugswettera@spardat.at> — 2001-10-01T14:39:36Z

   > IMHO, you should copy into a temporary table and the do a select 
   > distinct from it into the table that you want.
   
   Which would be way too slow for normal operation :-(
   We are talking about a "fast as possible" data load from a flat file
   that may have duplicates (or even data errors, but that 
   is another issue).
   
   Andreas
   
   

2. Re: Bulkloading using COPY - ignore duplicates?

   Thomas Swan <tswan@olemiss.edu> — 2001-10-01T14:42:56Z

   Zeugswetter Andreas SB SD wrote:
   
   >>IMHO, you should copy into a temporary table and the do a select 
   >>distinct from it into the table that you want.
   >>
   >
   >Which would be way too slow for normal operation :-(
   >We are talking about a "fast as possible" data load from a flat file
   >that may have duplicates (or even data errors, but that 
   >is another issue).
   >
   >Andreas
   >
   Then the IGNORE_DUPLICATE would definitely be the way to go, if speed is 
   the question...
   
   
   
   

3. Re: Bulkloading using COPY - ignore duplicates?

   Lee Kindness <lkindness@csl.co.uk> — 2001-10-01T15:31:06Z

   Okay,
   
   If I'm going to modify 'COPY INTO' to include 'ignore duplicates'
   functionality it looks like I'll have to add to the COPY syntax. The
   most obvious way is to add:
   
    WITH IGNORE DUPLICATES
   
   to the syntax. I'm going to need my hand held a bit for this! The
   grammar for COPY will need updating in gram.y and specifically the
   'WITH' keyword will have 'IGNORE DUPLICATES' as well as 'NULL AS'.
   
   Any pointers?
   
   Thanks, Lee.
   
   

4. Re: Bulkloading using COPY - ignore duplicates?

   Lee Kindness <lkindness@csl.co.uk> — 2001-10-01T15:50:41Z

   Lee Kindness writes:
    > If I'm going to modify 'COPY INTO' to include 'ignore duplicates'
    > functionality it looks like I'll have to add to the COPY syntax. The
    > most obvious way is to add:
    >  WITH IGNORE DUPLICATES
   
   Or does it make more sense to add a 'COPY_IGNORE_DUPLICATES' SET
   parameter? 
   
   Lee.