Thread

  1. BUG #19348: Disk space error when running COPY after upgrading to PostgreSQL 17.7

    PG Bug reporting form <noreply@postgresql.org> — 2025-12-09T11:04:21Z

    The following bug has been logged on the website:
    
    Bug reference:      19348
    Logged by:          weijie JL
    Email address:      weijie1006jl@gmail.com
    PostgreSQL version: 17.7
    Operating system:   RockyLinux8
    Description:        
    
    After upgrading to PostgreSQL 17.7, our scheduled data synchronization task
    intermittently fails. About 6 seconds into the run, during the COPY import
    stage, we get the following error:
    
    ERROR: 53100: could not extend file "base/16416/4201185432" with
    FileFallocate(): No space left on device
    However, the disk has ample free space. We see no abnormal data size growth,
    and the table involved is small — roughly a dozen columns, each containing
    short text strings of only a few dozen characters.
    
    The task runs every minute, inserting between several thousand and ~20,000
    rows. Failures happen sporadically, and the same task will usually succeed
    on the next scheduled run. Today we observed 6–7 failures.
    
    Stack trace logs have been collected and are attached here.
    2025-12-09 14:30:13.539 CST 10.11.15.37(53901) pgdb_axs xas [425550]LOG:
    00000: execute <unnamed>: CREATE TABLE mbd_mo_req_line_eaf07 (LIKE
    mbd_mo_req_line)
    2025-12-09 14:30:13.539 CST 10.11.15.37(53901) pgdb_axs xas
    [425550]LOCATION:  exec_execute_message, postgres.c:2222
    2025-12-09 14:30:13.541 CST 10.11.15.37(53901) pgdb_axs xas [425550]LOG:
    00000: execute <unnamed>: ALTER TABLE xas.mbd_mo_req_line_eaf07 ADD
    fl_stg_seq_id int4 NOT NULL
    2025-12-09 14:30:13.541 CST 10.11.15.37(53901) pgdb_axs xas
    [425550]LOCATION:  exec_execute_message, postgres.c:2222
    2025-12-09 14:30:13.542 CST 10.11.15.37(53901) pgdb_axs xas [425550]LOG:
    00000: execute <unnamed>: create unique index fl_mbd_mo_req_line_eaf07_dbmli
    on xas.mbd_mo_req_line_eaf07(fl_stg_seq_id)
    2025-12-09 14:30:13.542 CST 10.11.15.37(53901) pgdb_axs xas
    [425550]LOCATION:  exec_execute_message, postgres.c:2222
    2025-12-09 14:30:15.404 CST 10.11.15.37(51527) pgdb_axs xas [410019]ERROR:
    53100: could not extend file "base/16416/4201185432" with FileFallocate():
    No space left on device
    2025-12-09 14:30:15.404 CST 10.11.15.37(51527) pgdb_axs xas [410019]HINT:
    Check free disk space.
    2025-12-09 14:30:15.404 CST 10.11.15.37(51527) pgdb_axs xas [410019]CONTEXT:
    COPY mbd_mo_req_line_eaf07, line 446
    2025-12-09 14:30:15.404 CST 10.11.15.37(51527) pgdb_axs xas
    [410019]LOCATION:  mdzeroextend, md.c:591
    2025-12-09 14:30:15.404 CST 10.11.15.37(51527) pgdb_axs xas
    [410019]BACKTRACE:
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527) COPY() [0x4ee563]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527)
    COPY(smgrzeroextend+0x3d) [0x86560d]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527) COPY() [0x8365b9]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527)
    COPY(ExtendBufferedRelBy+0x49) [0x836919]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527) COPY() [0x561343]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527)
    COPY(RelationGetBufferForTuple+0x298) [0x561bf8]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527)
    COPY(heap_multi_insert+0x254) [0x5526f4]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527)
    COPY(CopyFrom+0x103d) [0x647d3d]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527) COPY(DoCopy+0x331)
    [0x6467d1]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527)
    COPY(standard_ProcessUtility+0x42c) [0x86fc5c]
            /usr/pgsql-17/lib/pg_stat_statements.so(+0x5dbd) [0x7f0cfa6dadbd]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527) COPY() [0x86e4cf]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527) COPY() [0x86e603]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527)
    COPY(PortalRun+0x27b) [0x86eb5b]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527) COPY() [0x86adab]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527)
    COPY(PostgresMain+0x14c8) [0x86c598]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527) COPY() [0x8672b5]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527)
    COPY(postmaster_child_launch+0x8f) [0x7d850f]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527) COPY() [0x7dbe5c]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527)
    COPY(PostmasterMain+0xd18) [0x7dda78]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527) COPY(main+0x223)
    [0x50f843]
            /lib64/libc.so.6(__libc_start_main+0xe5) [0x7f0d0c6b97e5]
            postgres: axs002: xas pgdb_axs 10.11.15.37(51527) COPY(_start+0x2e)
    [0x50fe1e]
    2025-12-09 14:30:15.404 CST 10.11.15.37(51527) pgdb_axs xas
    [410019]STATEMENT:  copy
    xas.mbd_mo_req_line_eaf07(req_line_id,org_id,mo_id,mo_status,lot_number,mo_line_id,line_lot_number,req_header_id,req_status,bom_version,item_id,unit_qty,req_qty,issued_qty,completed_qty,exceed_qty,is_exceed,original_req_line_id,is_lock,req_line_status,component_sequence_id,source_line_id,upd_time,upd_user_name,crt_time,crt_user_name,factory_id,mtr_ready_time,supply_type,operation_seq_no,parent_item_id,complete_replace,bin_value,item_supply_mode,produce_loss_qty,un_kitting_qty,created_order_qty,kitting_qty,supply_org_id,final_org_id,fl_stg_seq_id)from
    stdout delimiter ',' null as '\N' encoding 'UTF8'
    2025-12-09 14:30:15.407 CST 10.11.15.37(53901) pgdb_axs xas [425550]LOG:
    00000: execute <unnamed>: DROP TABLE IF EXISTS xas.mbd_mo_req_line_eaf07
    2025-12-09 14:30:15.407 CST 10.11.15.37(53901) pgdb_axs xas
    [425550]LOCATION:  exec_execute_message, postgres.c:2222
    
    more information:
    今天偶发有数据库报错,空间不足,但是磁盘空间是够的。
    [postgres@p-axs-d-002 log]$ df -h
    Filesystem               Size  Used Avail Use% Mounted on
    devtmpfs                 504G     0  504G   0% /dev
    tmpfs                    504G   40M  504G   1% /dev/shm
    tmpfs                    504G   19M  504G   1% /run
    tmpfs                    504G     0  504G   0% /sys/fs/cgroup
    /dev/mapper/rl-root      205G  9.8G  196G   5% /
    /dev/sda2               1014M  194M  821M  20% /boot
    /dev/sda1                599M  5.8M  594M   1% /boot/efi
    /dev/mapper/datavg-data   14T  9.0T  4.4T  68% /data
    tmpfs                    101G     0  101G   0% /run/user/0
    tmpfs                    101G     0  101G   0% /run/user/1001
    [postgres@p-axs-d-002 log]$ df -i
    Filesystem                  Inodes IUsed      IFree IUse% Mounted on
    devtmpfs                 131928720  1016  131927704    1% /dev
    tmpfs                    131933718   118  131933600    1% /dev/shm
    tmpfs                    131933718  1402  131932316    1% /run
    tmpfs                    131933718    17  131933701    1% /sys/fs/cgroup
    /dev/mapper/rl-root      107476992 80624  107396368    1% /
    /dev/sda2                   524288    19     524269    1% /boot
    /dev/sda1                        0     0          0     - /boot/efi
    /dev/mapper/datavg-data 1425278528 29497 1425249031    1% /data
    tmpfs                    131933718     6  131933712    1% /run/user/0
    tmpfs                    131933718     5  131933713    1% /run/user/1001
    
    [root@p-axs-d-002 data]# mount
    sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    devtmpfs on /dev type devtmpfs
    (rw,nosuid,seclabel,size=527714880k,nr_inodes=131928720,mode=755)
    securityfs on /sys/kernel/security type securityfs
    (rw,nosuid,nodev,noexec,relatime)
    tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
    devpts on /dev/pts type devpts
    (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
    tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
    tmpfs on /sys/fs/cgroup type tmpfs
    (ro,nosuid,nodev,noexec,seclabel,mode=755)
    cgroup on /sys/fs/cgroup/systemd type cgroup
    (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
    pstore on /sys/fs/pstore type pstore
    (rw,nosuid,nodev,noexec,relatime,seclabel)
    efivarfs on /sys/firmware/efi/efivars type efivarfs
    (rw,nosuid,nodev,noexec,relatime)
    bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
    cgroup on /sys/fs/cgroup/devices type cgroup
    (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
    cgroup on /sys/fs/cgroup/blkio type cgroup
    (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
    cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup
    (rw,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)
    cgroup on /sys/fs/cgroup/freezer type cgroup
    (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
    cgroup on /sys/fs/cgroup/pids type cgroup
    (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
    cgroup on /sys/fs/cgroup/perf_event type cgroup
    (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
    cgroup on /sys/fs/cgroup/memory type cgroup
    (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
    cgroup on /sys/fs/cgroup/hugetlb type cgroup
    (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
    cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup
    (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)
    cgroup on /sys/fs/cgroup/rdma type cgroup
    (rw,nosuid,nodev,noexec,relatime,seclabel,rdma)
    cgroup on /sys/fs/cgroup/cpuset type cgroup
    (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
    none on /sys/kernel/tracing type tracefs (rw,relatime,seclabel)
    configfs on /sys/kernel/config type configfs (rw,relatime)
    /dev/mapper/rl-root on / type xfs
    (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=128,swidth=128,noquota)
    selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
    mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
    hugetlbfs on /dev/hugepages type hugetlbfs
    (rw,relatime,seclabel,pagesize=2M)
    systemd-1 on /proc/sys/fs/binfmt_misc type autofs
    (rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=80565)
    debugfs on /sys/kernel/debug type debugfs (rw,relatime,seclabel)
    /dev/sda2 on /boot type xfs
    (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=128,swidth=128,noquota)
    /dev/sda1 on /boot/efi type vfat
    (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)
    /dev/mapper/datavg-data on /data type xfs
    (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=256,swidth=256,noquota)
    sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
    tmpfs on /run/user/0 type tmpfs
    (rw,nosuid,nodev,relatime,seclabel,size=105546972k,mode=700)
    tmpfs on /run/user/1001 type tmpfs
    (rw,nosuid,nodev,relatime,seclabel,size=105546972k,mode=700,uid=1001,gid=1001)