Reduce memory usage of tsvector type analyze function.
Heikki Linnakangas <heikki.linnakangas@iki.fi>
Reduce memory usage of tsvector type analyze function. compute_tsvector_stats() detoasted and kept in memory every tsvector value in the sample, but that can be a lot of memory. The original bug report described a case using over 10 gigabytes, with statistics target of 10000 (the maximum). To fix, allocate a separate copy of just the lexemes that we keep around, and free the detoasted tsvector values as we go. This adds some palloc/pfree overhead, when you have a lot of distinct lexemes in the sample, but it's better than running out of memory. Fixes bug #14654 reported by James C. Reviewed by Tom Lane. Backport to all supported versions. Discussion: https://www.postgresql.org/message-id/20170514200602.1451.46797@wrigleys.postgresql.org
Files
| Path | Change | +/− |
|---|---|---|
| src/backend/tsearch/ts_typanalyze.c | modified | +17 −4 |
Discussion
- BUG #14654: With high statistics targets on ts_vector, unexpectedly high memory use & OOM are triggered 4 messages · 2017-05-14 → 2017-07-12