Thread

1. expand on_error ignore error handling scope

   jian he <jian.universality@gmail.com> — 2025-04-30T00:00:00Z

   hi.
   
   Currently, (on_error ignore) only handles data type incompatibility errors.
   However, we can extend its functionality to also handle errors caused by extra
   data (additional columns) or missing data (fewer columns), I think.
   
   on_error is only applicable to non-binary formats.
   In non-binary formats (such as text and CSV), COPY FROM cut income source row by
   row by newline.  This means that row boundaries are clearly defined.  If a
   line's contents cannot be successfully converted to the table's columns, an
   error is raised.
   
   Based on this, I think it's doable to let on_error ignore handling missing and
   extra data errors.
   Hope I didn't miss anything.
   
   previously feature request discussion: [1]
   [1]: https://postgr.es/m/CAEHBEOCqqkSitU5gx%3D8aHD3ZeLtRy-1eC%3DXs%3DxB-thFNR9JpYA%40mail.gmail.com
   
   what do you think?
   
   
   
   

2. Re: expand on_error ignore error handling scope

   Michael Paquier <michael@paquier.xyz> — 2025-04-30T01:03:12Z

   On Wed, Apr 30, 2025 at 08:00:00AM +0800, jian he wrote:
   > Currently, (on_error ignore) only handles data type incompatibility errors.
   > However, we can extend its functionality to also handle errors caused by extra
   > data (additional columns) or missing data (fewer columns), I think.
   
   The original argument behind type incompatibility checks was that
   incorrectly shaped multi-byte utf-8 character sequences could be
   problematic, particularly if these are cut with a newline.  It should
   take care of most cases I've heard about (these should be arguments
   about that around pg_bulkload, as well).
   
   > on_error is only applicable to non-binary formats.
   > In non-binary formats (such as text and CSV), COPY FROM cut income source row by
   > row by newline.  This means that row boundaries are clearly defined.  If a
   > line's contents cannot be successfully converted to the table's columns, an
   > error is raised.
   
   That depends.  If the code can be made more verbose with more details
   depending on the error path reached without blowing up the whole COPY,
   knowing more about the incorrect state of a row without extending
   ON_ERROR with a new mode seems useful to me.
   --
   Michael
   

3. Re: expand on_error ignore error handling scope

   torikoshia <torikoshia@oss.nttdata.com> — 2025-06-05T02:15:40Z

   On 2025-04-30 09:00, jian he wrote:
   > hi.
   > 
   > Currently, (on_error ignore) only handles data type incompatibility 
   > errors.
   > However, we can extend its functionality to also handle errors caused 
   > by extra
   > data (additional columns) or missing data (fewer columns), I think.
   > 
   > on_error is only applicable to non-binary formats.
   > In non-binary formats (such as text and CSV), COPY FROM cut income 
   > source row by
   > row by newline.  This means that row boundaries are clearly defined.  
   > If a
   > line's contents cannot be successfully converted to the table's 
   > columns, an
   > error is raised.
   > 
   > Based on this, I think it's doable to let on_error ignore handling 
   > missing and
   > extra data errors.
   > Hope I didn't miss anything.
   > 
   > previously feature request discussion: [1]
   > [1]:
   > https://postgr.es/m/CAEHBEOCqqkSitU5gx%3D8aHD3ZeLtRy-1eC%3DXs%3DxB-thFNR9JpYA%40mail.gmail.com
   > 
   > what do you think?
   
   I've heard similar feedback from users who were surprised that setting 
   on_error = ignore didn't suppress errors due to column count mismatches.
   Supporting this case as well might make the feature more intuitive and 
   helpful for users.
   
   That said, there might be cases where users actually want column count 
   mismatches to result in an error, even when on_error is set to ignore.
   So perhaps it could be worth considering -- just as an idea -- whether 
   we should allow more fine-grained control over which kinds of errors to 
   ignore, e.g., adding a new option to what type of errors are ignored.
   
   -- 
   Regards,
   
   --
   Atsushi Torikoshi
   Seconded from NTT DATA GROUP CORPORATION to SRA OSS K.K.