CrunchyData Blog

Postgres 19: How Our Advice Has Changed Since We Wrote It

Christopher.Winslett@crunchydata.com (Christopher Winslett) — Tue, 18 Aug 2026 15:00:00 EDT

Over the years we have written a lot about how data gets into Postgres, how it sits on disk, and how indexes help you find it again. Some of that advice was written against Postgres 10 or 11. A surprising amount of it is still exactly what we would tell you for the upcoming Postgres 19 release. Functionality described here is based on current betas; minor details may still change before GA.

This post revisits Crunchy posts in the “load, storage, indexes, and partitioning” bucket: what we wrote, which version moved the needle, and what we would tell you to do now. Along the way: async I/O, more resilient COPY, LZ4 by default, richer BRIN shapes, skip scan, and smoother partition operations.

Async I/O: faster scans and vacuum on modern storage

In 2019 we benchmarked a BRIN index against a B-tree and a parallel sequential scan on the same time-series table. Sometimes BRIN won. Sometimes the parallel seq scan won: four workers chewing through the heap beat a clever index. That was the right lesson for Postgres 11: indexes are a tradeoff against what the executor can already do in parallel.

Postgres 18 made those heap reads substantially faster.

Async I/O lets backends queue multiple disk reads instead of waiting on each one. Sequential scans, bitmap heap scans (the path BRIN and many bitmap index plans finish with), and vacuum all benefit. Community benchmarks have shown up to ~3× on cold, latency-bound storage, a big deal for cloud disks. Defaults matter here: io_method = worker is on out of the box; on Linux 5.1+ you can try io_method = io_uring. See Get Excited About Postgres 18 for the operator view.

Postgres 19 builds on that: I/O workers can autoscale (io_min_workers / io_max_workers), read-ahead scheduling improved, and EXPLAIN (ANALYZE, IO) can show what the async subsystem is doing. Parallel query is still there; each worker can queue several reads and keep making progress while some of them are still in flight, so you get more useful work between waits. Parallel autovacuum workers also landed in 19 (autovacuum_max_parallel_workers and per-table autovacuum_parallel_workers), so maintenance can fan out, but the defaults are conservative: tune them when vacuum is falling behind on large tables.

One related default flip: JIT is off by default in Postgres 19 (it had been on since 12). The old cost model was unreliable, so large analytical or parallel scans that used to compile at runtime no longer will unless you turn jit back on. If those workloads matter to you, re-enable explicitly and re-check plans after upgrade.

Postgres 19 advice: Keep choosing indexes for selectivity, and re-test BRIN-vs-parallel plans on your storage with EXPLAIN (ANALYZE, BUFFERS, IO) after you upgrade. Tune effective_io_concurrency / maintenance_io_concurrency with the new defaults in mind (they rose to 16 in 18). Treat async I/O as more headroom for the heap paths, alongside good COPY, TOAST modeling, BRIN, and covering indexes.

Loading data: COPY is still king, and now more resilient

What we wrote: In Fast CSV and JSON Ingestion in PostgreSQL with COPY (2018, Postgres 10), Jonathan Katz showed the classic pattern: generate CSV or newline-delimited JSON, pipe it into psql, and let COPY ... FROM STDIN do the heavy lifting. Prefer COPY over row-by-row INSERT. Store JSON as jsonb. Add a GIN index when you need containment queries.

That advice was right then and is still right now. The load path itself did not need reinventing; it got more capable.

What changed:

Version	Change that matters for loads
16	`COPY FROM` can map a sentinel string to a column `DEFAULT`
17	`ON_ERROR ignore` skips bad type conversions and keeps loading; `LOG_VERBOSITY` reports what was skipped
18	`REJECT_LIMIT` caps how many bad rows you will tolerate; `LOG_VERBOSITY silent` quiets the noise; CSV handling of `\.` is clearer
19	Faster text/CSV parsing via SIMD; `ON_ERROR SET_NULL` turns invalid values into NULL; skip multiple header lines; `COPY TO` can emit JSON (and a single JSON array with `FORCE_ARRAY`) and can target partitioned tables directly

FREEZE on an initial load into a freshly created or truncated table is still the right performance trick when you want to skip a later freeze vacuum. That option predated these releases and remains useful.

Postgres 19 advice: Keep using COPY for bulk ingest, including imperfect feeds. A practical 19-era load looks like this:

COPY events (event_id, occurred_at, payload)
FROM STDIN
WITH (
  FORMAT csv,
  HEADER 2,
  DEFAULT '__DEFAULT__',
  ON_ERROR set_null,
  LOG_VERBOSITY verbose
);

HEADER 2 skips two lead-in lines (title row plus column names, or whatever your export prepends). ON_ERROR set_null keeps the row and nulls only the bad field instead of discarding the whole line. Prefer ON_ERROR ignore when a bad cell should drop the row entirely; pair that with REJECT_LIMIT when you want a hard cap on how many skips you will tolerate:

COPY events (event_id, occurred_at, payload)
FROM STDIN
WITH (
  FORMAT csv,
  HEADER 2,
  DEFAULT '__DEFAULT__',
  ON_ERROR ignore,
  REJECT_LIMIT 1000,
  LOG_VERBOSITY verbose
);

Here DEFAULT '__DEFAULT__' means: when the CSV cell is the literal string __DEFAULT__, use the column’s default expression. Pick any sentinel that will not appear as real data. (Avoid '\N' in CSV: readers often confuse it with the NULL marker.) Confirm final GA docs for edge cases around DEFAULT, ON_ERROR, and REJECT_LIMIT once 19 ships.

Prefer FREEZE for empty staging tables that you load once and then promote. And when you need to export partitioned data or hand JSON to another system, COPY TO handles the parent directly:

COPY events TO STDOUT WITH (FORMAT json);
COPY events TO STDOUT WITH (FORMAT json, FORCE_ARRAY);

Plain FORMAT json streams newline-delimited JSON (NDJSON/JSONL: one object per line), which is handy for batch pipelines. FORCE_ARRAY wraps the whole result in a single JSON array ([...]), which some web APIs expect.

The 2018 post’s JSON tip still stands: ingest into jsonb, not text json, unless you have a specific reason to preserve exact formatting.

TOAST: same mechanism, faster default compression

What we wrote: In Postgres TOAST: The Greatest Thing Since Sliced Bread? (2024), Elizabeth Christensen walked through the 8 kB page, the ~2 kB toast_tuple_target, storage strategies (PLAIN / EXTENDED / EXTERNAL / MAIN), and the practical guidance: updating a toasted row rewrites toast chunks, frequent access of compressed values has a cost, and large JSON or text that you query often belongs in a better data model, not in a giant toasted blob.

That mental model is still the right one. TOAST did not get replaced; it got a better default compressor.

What changed:

Version	Change
14	LZ4 becomes available via `default_toast_compression = 'lz4'` or per-column `COMPRESSION lz4`
19	LZ4 becomes the default (`default_toast_compression` flips from `pglz` to `lz4`); native `REPACK` (with `CONCURRENTLY`) rewrites tables without a long exclusive lock

We covered the compression decision tree in more depth in Postgres 19 Compression: from pglz to LZ4. The short version: LZ4 compresses and decompresses much faster than pglz, with a similar compression ratio, and still fails fast on incompressible data.

Postgres 19 advice: Keep Elizabeth’s modeling advice. Prefer structured columns for hot lookup data when you can. Prefer EXTERNAL when you have already compressed the payload yourself. On a fresh Postgres 19 cluster, LZ4 is what new toasted values get unless you override it, with no extra config step required.

After an upgrade from an older major version, existing toast values stay in whatever algorithm compressed them. New writes follow the new default. If you want to measure the win:

SELECT
  pg_column_size(payload) AS stored_bytes,
  octet_length(payload) AS raw_bytes
FROM events
WHERE length(payload) > 100
LIMIT 5;

Rebuild or rewrite only when storage or CPU on hot toasted columns is worth the effort. The default change already helps new data. When you do need a rewrite to reclaim bloat, Postgres 19’s native REPACK is the clean path; CONCURRENTLY keeps the table readable and writable while the new heap is built:

REPACK (CONCURRENTLY, ANALYZE) events;

CONCURRENTLY needs a primary key or index-based replica identity so Postgres can track and replay concurrent WAL updates, and it does not run on a partitioned parent (repack individual children instead). It also builds a full temporary copy of the relation, so plan on roughly 2× the table’s disk footprint while the rewrite runs. It is not MVCC-safe in the same narrow sense as TRUNCATE: a concurrent transaction that took a snapshot before the final swap and had not yet touched the table can briefly see it as empty. For blocking rewrites, plain REPACK still replaces the old VACUUM FULL / CLUSTER pair.

Neither plain nor concurrent REPACK recompresses existing toast values. After an upgrade, old pglz chunks stay pglz until those rows are updated (or you rewrite them some other way); new toasted writes follow the LZ4 default. Check with pg_column_compression() if you need to confirm.

BRIN: still tiny, richer opclasses, faster to build

What we wrote: In PostgreSQL BRIN Indexes: Big Data Performance With Minimal Storage (2019, Postgres 11), Jonathan showed BRIN crushing a B-tree on size for append-mostly sensor data: a 32 kB BRIN versus a 214 MB B-tree on the same timestamp column, with strong range-query performance when physical order and logical order lined up. Later posts (Avoiding the Pitfalls of BRIN Indexes and Paul Ramsey’s When Does BRIN Win?) sharpened when BRIN shines: high correlation with physical order. Random or shuffled layouts need a different shape of index.

What changed:

Version	Change
14	`minmax_multi` opclasses store multiple min/max values per range (great with outliers); Bloom BRIN opclasses work for equality on less-correlated data
17	Parallel `CREATE INDEX` for BRIN

The gains here are mostly from 14 and 17 (better opclasses and parallel builds). On 19, BRIN mostly inherits the surrounding I/O improvements: async reads speed the bitmap heap scan that follows a BRIN hit. The default single min/max summarizer remains a strong choice for clean append-only time series; when outliers show up, minmax_multi keeps ranges useful by tracking several min/max pairs per block range instead of one stretched span.

Postgres 19 advice: Start with Paul’s question: is this column correlated with physical order? If yes, use BRIN. On 14+, prefer multi-minmax when the table is large and not perfectly ordered:

CREATE INDEX scans_created_at_brin_idx
  ON scans
  USING brin (created_at timestamptz_minmax_multi_ops)
  WITH (pages_per_range = 32);

Tune pages_per_range to your typical filter width. The default is 128 (about 1 MB of heap per summary). 32 raises index resolution and tightens heap probes at the cost of a slightly larger (still tiny) index, which helps when time-series inserts have mild out-of-order jitter. For equality lookups on values that are not well ordered on disk (UUIDs, MAC addresses, some categorical keys), try a Bloom BRIN opclass, often enough without a full-size B-tree:

CREATE INDEX events_id_brin_bloom
  ON events
  USING brin (event_id uuid_bloom_ops);

On Postgres 17+, BRIN builds on large tables can use parallel workers automatically when maintenance_work_mem and parallel settings allow it. You do not need a special syntax: give the build enough memory and let it fan out.

And remember the 2019 twist ending: sometimes a parallel sequential scan beat BRIN. On 18+, async I/O speeds both the parallel seq scan and the bitmap heap scan that follows a BRIN hit. Re-run the bake-off on your storage, especially cloud disks, and pick the winner with fresh numbers.

BRIN is still the specialist for correlated, append-mostly data, not a general-purpose B-tree replacement. The upgrade is that you have more shapes to choose from: classic minmax, multi-minmax, and Bloom, plus parallel builds.

Covering indexes: INCLUDE still helps, and skip scan covers more queries

What we wrote: In Why Covering Indexes in Postgres Are Incredibly Helpful (2018, Postgres 11), Jonathan introduced B-tree INCLUDE columns so queries could satisfy SELECT lists from the index alone, including types that cannot be B-tree keys, like point. Vacuum the table so the visibility map allows true index-only scans. Be conservative: every included column costs write amplification and disk.

That tradeoff is unchanged. INCLUDE is still how you build a covering index without polluting the sort key.

What changed:

Version	Change
17	More efficient B-tree scans for multi-value lookups such as `IN (...)`
18	B-tree skip scan: a multicolumn index can be used when equality predicates hit later columns and omitted leading columns have low cardinality

Skip scan is the big unlock. An index on (visitor, visited_at) can now help more queries that filter on later columns when the leading column has low cardinality. Teams can often keep one well-designed composite index instead of adding a companion for every access pattern. We covered the mechanics in Get Excited About Postgres 18.

Postgres 19 advice: Still use INCLUDE when you know a hot query’s filter columns and its select list:

CREATE UNIQUE INDEX visits_visitor_visited_at_geocode_idx
  ON visits (visitor, visited_at)
  INCLUDE (geocode);

Still VACUUM (or let autovacuum catch up) before you expect Heap Fetches: 0.

For your other indexes, check whether skip scan already covers a query before you add another index, especially when the leading column is something like status, region, or tenant with a small number of distinct values. If an equality-only single-column index exists only to cover those “missing leading column” cases, try dropping it after you confirm the composite index serves the workload. Keep dedicated leading-key indexes for high-cardinality first columns and for inequality-driven access patterns skip scan cannot help.

Also lean on Postgres 17’s bulk-scan improvements: WHERE id IN (...) style lookups on B-trees got cheaper without any schema change. Re-benchmark before you invent a custom “optimize IN lists” pattern an upgrade already improved.

Partitioning: still for lifecycle, now smoother to operate

What we wrote: In Partitioning with Native Postgres and pg_partman (2022), Elizabeth Christensen framed partitioning as a scale and lifecycle tool: archive old ranges by detaching, create tomorrow’s child, let pg_partman handle the calendar. Prefer native declarative partitioning over trigger/inheritance, advice that goes back to Keith Fiske’s migrate-from-trigger-based guide (2020). Later posts sharpened the edges: default partitions as a safety net you monitor and drain, and retention with pg_partman so dropping a child returns space immediately instead of waiting on DELETE + vacuum.

The 2022 tutorial’s detach looked like this:

ALTER TABLE iot_thermostat DETACH PARTITION iot_thermostat07142022;

That still works. On Postgres 14+, you can do the same rotation online with CONCURRENTLY. The caveats section correctly noted that unique indexes generally must include the partition key. That part is still true.

What changed:

Version	Change
14	`DETACH PARTITION ... CONCURRENTLY`: online detach with lighter locking
15	Cleaner FK behavior when an `UPDATE` moves a row between partitions
17	Identity columns on partitioned tables; exclusion constraints when they equality-compare the partition key
18	`VACUUM` / `ANALYZE ... ONLY` on the parent; `NOT VALID` FKs on partitioned tables
19	Native `MERGE PARTITIONS` / `SPLIT PARTITION`; `COPY TO` works on partitioned parents; `vacuumdb --analyze-only` analyzes partitioned parents by default

Planner and pruning improvements landed steadily from 12 through 18, so larger partition counts plan more efficiently than they did in the early native era. You still want a practical partition size (day, month, quarter), but you have more leeway than early native partitioning allowed.

Postgres 19 advice: Keep partitioning for retention and operational boundaries first. Query speed is a bonus when predicates hit the partition key, not the reason to partition a 200 GB table.

For day-to-day rotation, prefer concurrent detach:

ALTER TABLE iot_thermostat
  DETACH PARTITION iot_thermostat07142022 CONCURRENTLY;

Keep using a default partition to catch bad timestamps and late-arriving keys, then move that data into real children. Keith’s default-partition guidance still applies. On 17+, you can put identity columns on the partitioned parent directly.

When you need to coarsen history (monthly → quarterly) or split a hot range, Postgres 19 can do it in SQL:

ALTER TABLE events
  MERGE PARTITIONS (events_2024_01, events_2024_02, events_2024_03)
  INTO events_2024_q1;

ALTER TABLE events
  SPLIT PARTITION events_2024_q1 INTO (
    PARTITION events_2024_01 FOR VALUES FROM ('2024-01-01') TO ('2024-02-01'),
    PARTITION events_2024_02 FOR VALUES FROM ('2024-02-01') TO ('2024-03-01'),
    PARTITION events_2024_03 FOR VALUES FROM ('2024-03-01') TO ('2024-04-01')
  );

Plan merge/split for a maintenance window today: they take ACCESS EXCLUSIVE on the parent and do not yet offer CONCURRENTLY. Under the hood they physically copy tuples into new child relations, so that exclusive lock lasts for the entire copy, not just a metadata swap. On high-throughput tables, prefer concurrent detach / attach patterns for zero-downtime rotation and reserve merge/split for quieter windows. Also budget for catalog work: objects defined directly on the source partitions (partition-local indexes and constraints) are dropped during the split or merge. Indexes and constraints defined on the parent are recreated on the new children automatically. For online work, concurrent detach (or attach a pre-built child) remains the smooth path.

Unique indexes still need the partition key if you want uniqueness across the set. pg_partman remains the right automation layer for create/retain schedules. Native DDL got richer, and the calendar automation still complements it.

Exporting a partitioned table is simpler too. Before Postgres 19, COPY parent TO ... failed with cannot copy from partitioned table, so you had to wrap it as COPY (SELECT * FROM parent) TO ... or target children one by one. In 19, COPY TO runs directly on the root relation and pulls rows from all descendants, which skips the extra planning and executor overhead of the SELECT wrapper and keeps bulk exports cheaper on CPU and memory.

That same path helps logical replication’s initial table sync. When you publish a partitioned parent with publish_via_partition_root = true, snapshot copy workers on the publisher use COPY TO. In 19 they can take the native direct parent path instead of the query-wrapper workaround, which streamlines the initial streaming sync.

What hasn’t changed

After all those version notes, here is the reassuring part. These lasting Crunchy recommendations (ideas we have been writing about for years) remain correct on Postgres 19:

Prefer COPY over row-by-row INSERT for bulk loads. Streaming CSV or newline-delimited JSON into COPY ... FROM STDIN is still the fast path.
Store JSON as jsonb, not text json, unless you specifically need to preserve exact formatting. Index with GIN when you need containment queries.
Keep hot data out of TOAST when you can. Large blobs you filter and join on belong in structured columns; TOAST is for oversized attributes, not your primary access path.
Indexes are a tradeoff, not free. They speed lookups and can help joins and sorts, and they cost disk, write amplification, and maintenance.
The planner may skip your index. Creating an index does not guarantee it will be used; verify with EXPLAIN (ANALYZE, BUFFERS).
B-tree is the default workhorse. Reach for other index types when the access pattern calls for them, not because B-tree is outdated.
BRIN wins on correlated, append-mostly columns. Tiny indexes for time-series and sensor-style data when physical order matches logical order.
Covering indexes (INCLUDE) enable index-only scans when the select list fits in the index after vacuum has updated the visibility map. Don’t over-include wide columns.
Partition for lifecycle and cheap drops first. Retention by detaching or dropping children beats giant DELETEs; query speed is a bonus when predicates hit the partition key.
Unique indexes on partitioned tables need the partition key if you want uniqueness across the whole set. Native declarative partitioning beats trigger/inheritance routing.
Use a default partition as a safety net, and drain it. Catch bad or late keys there, then move valid rows into real children.
pg_partman still handles the schedule. Native DDL got richer; automatic create/retain schedules are still an extension’s job.

The advice did not flip. Postgres got better at delivering it: more resilient loads, faster compression by default, more BRIN shapes, wider B-tree usefulness, quicker disk reads, and smoother partition operations. If you still follow those older posts on Postgres 19, keep the mental model, update the edges of the checklist, and re-test after upgrade with EXPLAIN (ANALYZE, BUFFERS, IO).

Postgres 19 Compression: from pglz to LZ4

Christopher.Winslett@crunchydata.com (Christopher Winslett) — Thu, 16 Jul 2026 08:00:00 EDT

UPDATE: For how the LZ4 default fits into the broader TOAST modeling advice—and related Postgres 19 changes around COPY, BRIN, covering indexes, and partitioning—see Postgres 19: How Our Advice Has Changed Since We Wrote It.

Postgres 19 is planning to change the default TOAST compression from pglz to LZ4, so let's look at how Postgres compresses data in table storage and indexes. Postgres uses a single, unified compression framework for table (heap), TOAST, and indexes. In heap and TOAST, compression is automatic and on by default for variable-length types like TEXT, VARCHAR, BYTEA, and JSONB. In indexes, it is opportunistic: compression fires only when an individual key exceeds the size threshold, not for every variable-length value stored in an index.

click to expand

History of Postgres Compression

Compression was first added in Postgres 7.0, released in 2000, but not in the form it takes today. At the time, Postgres had a strict 8kB maximum row size, and trying to insert more than 8kB would throw an error. Fixing this row size limit was a priority for the Postgres core team.

The first attempt to work around the limit was an explicitly compressed field. Postgres 7.0 shipped an lztext data type that used the pglz compression algorithm. This implementation had tradeoffs: the 8kB row limit still existed, and users had to explicitly choose a compressed data type.

After the 7.0 release, the next logical step might have been to add more compressed data types like LONG or BLOB (as many closed source databases were doing at the time). Instead, the Postgres core team rejected additional data types and rallied around TOAST. Discussions on the pgsql-hackers mailing list show the group split the 8kB problem into two distinct problems: a data type problem and a physical storage problem. Compression and TOAST were the answer to the physical storage problem.

TOAST with pglz

With Postgres 7.1, TOAST was implemented using pglz.

The pglz algorithm lives in the pg_lzcompress.c file. Because Postgres is open source, we can read the author's reasoning for home-rolling a compression algorithm right in the comments:

Trade ratio for speed: pglz is fast to compress and decompress, and willingly gives up compression ratio to get there. The source code calls this another "speed against ratio" preference characteristic of the algorithm.
Minimal memory usage: the algorithm uses a 4096-byte sliding window, small enough not to affect Postgres' memory needs. The comments note that the compressor works best for attributes of a size between 1K and 1M, so it trades away performance on very large values.
Aggressive fail-safe: the algorithm is designed to fail fast when data does not compress well, to avoid wasting CPU cycles.
No external dependencies: the algorithm is self-contained and does not require any external libraries. Back then, Linux was not the default server OS for the cloud (nor was there a "cloud"), so Postgres needed a compression algorithm that would work everywhere it was compiled.

Jan Wieck authored the algorithm and left an acknowledgement at the bottom of the file:

Many thanks to Adisak Pochanayon, who's article about SLZ inspired me to write the PostgreSQL compression this way.

Adisak Pochanayon's SLZ article described a compression scheme built for the game industry, where it was used to compress graphics and audio data.

Why move from pglz to LZ4?

pglz was built for a different era, and LZ4 is a modern algorithm whose tradeoffs match modern hardware. Postgres' LZ4 rollout follows a path similar to the original pglz implementation: first as an option, then as standard. Since Postgres 14, LZ4 has been available via a system-wide setting (default_toast_compression = 'lz4') or a column-specific setting (column_name text COMPRESSION lz4).

Faster compression: LZ4 compresses significantly faster than pglz. On an totally unscientific test, a 2,000 row INSERT of ~10 kB values, LZ4 completes in roughly 6 ms versus 50 ms for pglz (8× faster). Decompression throughput is similar between the two algorithms with this data set, as I/O and memory latency dominated over CPU decode time.
Better compression ratio (sometimes): LZ4's 64 kB sliding window (versus pglz's 4 kB) finds more back-references in the data, which expands compression. On a similar, totally unscientific workload, LZ4 produced 111 bytes stored per 10,400 bytes raw (98.9% reduction) versus pglz's 186 bytes (98.2% reduction), and used 312 kB of total heap space versus pglz's 472 kB. There are situations where pglz has a better compression ratio.
Continues to fail fast: LZ4 has an efficient early-abort mechanism designed to detect random or incompressible data quickly.

Postgres storage strategies

First off, you need to know that Postgres has a few different storage strategies for columns:

EXTENDED (we capitalize it because the Postgres docs do, not because we're yelling) lets Postgres use every tool available. This is the default for variable-length types like TEXT, VARCHAR, BYTEA, and JSONB. Values can be stored uncompressed, compressed, in the heap, or in TOAST.

PLAIN stores the column inline in the heap, uncompressed. It is the default for fixed-width types like INT, FLOAT, and BOOL. These values rarely benefit from compression.

EXTERNAL tells Postgres to store the column in TOAST, but not compress it.

MAIN tells Postgres to try compressing the column, but avoid moving it to TOAST if possible.

The varlena format

For variable-length types, Postgres uses a format called varlena to store the data. varlena is a self-describing format with a header that records the length of the data and whether it is compressed. It is used for all variable-length types (including TEXT, VARCHAR, BYTEA, and JSONB) and it is what gets stored in the heap, in TOAST, and in indexes.

Only variable-length types are compressed. Fixed-length types (like INT, FLOAT, and BOOL) are never compressed; they are stored directly in the heap.

The compression decision tree

When writing data (insert or update), Postgres attempts to get the row size below a threshold of roughly 2 kB (toast_tuple_target, default 2040 bytes). It uses the following decision tree for EXTENDED columns:

If the entire row is already smaller than ~2 kB, Postgres writes the row directly to the heap, uncompressed.
If the row exceeds the threshold, Postgres sorts the EXTENDED columns by size and attempts to compress the largest one. If compression succeeds and brings the total row size under the threshold, it stops and writes to the heap. If not, it moves to the next largest column.
If all eligible columns are compressed and the row still exceeds the threshold, Postgres begins moving the largest EXTENDED or EXTERNAL columns out-of-line into the TOAST table, replacing each with an 18-byte pointer in the main heap tuple until the row fits.
If it still doesn't fit, Postgres loops back to compress any MAIN columns inline. If that fails, it takes the last resort: moving those MAIN columns out-of-line into TOAST.

For more on TOAST, check out Postgres TOAST: The Greatest Thing Since Sliced Bread?

Checking actual compression savings

-- pg_column_size: bytes as stored in the heap or TOAST table (after compression)
-- octet_length:  bytes of raw character data (uncompressed)

SELECT
  pg_column_size(payload) AS stored_bytes,
  octet_length(payload)   AS raw_bytes,
  round(
    (1 - pg_column_size(payload)::numeric
          / NULLIF(octet_length(payload), 0)) * 100, 1
  ) AS compression_pct
FROM events WHERE length(payload) > 100 LIMIT 5;

pg_column_size reports the compressed data size. When it returns a value much smaller than octet_length, the value was successfully compressed. When the two are approximately equal, the value did not compress well enough to save space. In that case, if the value is large (above the ~2 kB threshold), it will still have been moved to the TOAST table uncompressed.

Compression in indexes

B-tree index pages store key values as IndexTuple entries. Each entry contains an IndexTupleData header (8 bytes) followed by the key datum. For variable-length types, the datum uses the same varlena format as heap tuples (piggybacking on the workaround built for the 8kB row limit).

Postgres uses a single compression framework tied to the TOAST architecture. It flags in the varlena header whether the data is compressed. If the heap already compressed a value, it goes into the index compressed. If a value is uncompressed and exceeds 510 bytes (TOAST_INDEX_TARGET, about 1/16 of the 8 kB buffer page), the index code invokes the same TOAST compression routine inline to try to make it fit. If the compressed form fits, it is stored compressed. If even the compressed form exceeds the limit, the write transaction fails.

This is why repeat('x', 5000) can be B-tree indexed: LZ4 compresses 5,000 repeated characters down to ~38 bytes, well within the 2704-byte cap. Random or pseudo-random data of the same length produces a compressed form nearly equal to the original, which exceeds the cap and cannot be indexed.

-- These succeed: both compressible, both fit after LZ4 compression
CREATE INDEX ON docs (body);

INSERT INTO docs VALUES (repeat('x', 5000));    -- stored: ~38 bytes compressed
INSERT INTO docs VALUES (repeat('ab', 2000));   -- stored: ~35 bytes compressed

-- This fails: md5 output is pseudo-random, essentially incompressible
INSERT INTO docs VALUES (
  (SELECT string_agg(md5(g::text), '') FROM generate_series(1, 88) g)
);
-- 2816 chars of MD5 → compressed form ≈ 2816 bytes → index row size 2832 > 2704
-- ERROR:  index row size 2832 exceeds btree version 4 maximum 2704 for index "..."
-- HINT:   Values larger than 1/3 of a buffer page cannot be indexed.

The future of compression in Postgres

There's a lot to learn about how Postgres moves forward by looking at compression. The early false step of dedicated compressed data types was acknowledged, and the underlying pglz work was retooled into TOAST, which has been a huge success. In moving from pglz to LZ4, Postgres is taking a similar approach: first a test, then a migration. The core team has moved intentionally to make sure the compression algorithm change is the correct path.

British Columbia, Time Zones, and Postgres

Christopher.Winslett@crunchydata.com (Christopher Winslett) — Tue, 16 Jun 2026 08:00:00 EDT

On March 8, 2026, British Columbia moved their clocks to a year-round Pacific Daylight Savings Time. In March, they did the spring forward one hour with their clocks to UTC-7, but they won't fall back to UTC-8 in November. Going forward, the UTC offset for America/Vancouver timezone is permanently UTC-7.

Let's use this as an opportunity to talk about date and time zone storage. In the most basic examples, the default is to store the UTC value, then calculate local time relative to UTC. However, people using calendar systems think in terms of local time (i.e. wall clock time), and never consider UTC. After modifying time zone data, these time calculations from UTC for a region will differ from the user's input value.

If you stored timestamps in a UTC-based column for British Columbia-based appointment in 2026 and beyond, your November through March appointments may be off by an hour!

See timestamptz columns don't store the local time. They store the UTC time, and the timezone is only used to convert to and from UTC when inserting and querying. If you stored a future appointment as a timestamptz in the America/Vancouver timezone, it was converted to UTC using the rules at the time of storage. When you query that appointment later, it converts back to local time using the current rules. If the rules changed from storage to query, the local time you get back is not what the user originally intended.

If you've not updated your tzdata package, then Postgres doesn't know about the change, and it will continue to convert using the old rules. How often are the tzdata packages in Ubuntu updated? Surprisingly, every few months.

If your columns are stored in timestamptz column types and work with customers in British Columbia, use the following SQL query to determine if the tzdata package has been updated:

SELECT
  to_char(
    '2026-12-01 10:00:00'::timestamp AT TIME ZONE 'America/Vancouver',
    'HH24:MI:SS OF'
  ) AS november_2026_vancouver_offset;

If the value is 17:00:00 +00, then tzdata has been updated. This is not as good as it sounds because it will require digging through logs to know if future appointments were created before or after the the timezone adjustment.

If the value is 18:00:00 +00, then good news! Your tzdata has not been updated, and you do not have data split over the updates.

An Example of the Timezone Shift

Earlier this year, a user booked a 10 AM appointment for November 10, 2026 in Vancouver. You store it as a timestamptz:

INSERT INTO appointments (patient_id, starts_at)
VALUES (42, '2026-11-10T10:00:00-08:00');
-- stored as: 2026-11-10 18:00:00+00  (UTC)

In April 2026, the tzdata update is released to push the new timezone rules.

On November 10, 2026, the patient shows up at 10 AM local time as they documented in their calendar. But when you query the appointment, it says their appointment is at 11 AM local time:

SELECT starts_at AT TIME ZONE 'America/Vancouver' AS local_time
FROM appointments
WHERE patient_id = 42;
-- returns: 2026-11-10 11:00:00

Notice it is calculated as an hour later than originally entered.

A schema that survives time zone changes: dual column pattern

As its name implies, a dual-column pattern stores data in two columns (actually three):

local timestamp
local timezone
UTC timestamp

The UTC timestamp column should be a calculated column. Use the timestamp and timezone to calculate UTC. That calculated UTC value would also be stored and queried to enable background jobs to send notifications and simplify constraint checking, like appointment collisions.

The dual-column pattern is necessary when the local intent is authoritative: people or deliveries at a time and place, legal deadlines, calendar events, etc.

Don't go overboard though. When the event is in the past, or the exact UTC moment is authoritative (log entries, financial transactions, sensor readings), use plain timestamptz. The dual-column pattern adds cost and complexity only worth paying when future local intent must be preserved.

The detailed schema would look like this:

CREATE TABLE appointments (
  id             bigint      PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
  local_time     timestamp   NOT NULL,   -- wall clock value
  timezone_name  text        NOT NULL,   -- IANA name: 'America/Vancouver'
  starts_at_utc  timestamptz NOT NULL    -- Calculated via trigger
  ...
);

local_time and timezone_name together answer the "what did the user intend?" by storing the wall-calendar / wall-clock values / wall-clock location. These values should only change at the user's request. They will be used to calculate the starts_at_utc.

starts_at_utc can be the column you index, query, and use for constraints. It answers "what UTC moment does this appointment correspond to right now?" Having a calculated, stored UTC value should simplify using the UTC value as you currently do.

There are a few ways to calculate starts_at_utc, using an application or the database. While the calculated UTC column would be a great example of a generated column, Postgres doesn't allow timestamp with time zone column types for generated columns because timestamptz is not classified as immutable since timezone rules change. So, use a trigger to compute starts_at on insert and update:

CREATE OR REPLACE FUNCTION recompute_appointment_utc()
RETURNS TRIGGER AS $$
BEGIN
  NEW.starts_at_utc := NEW.local_time AT TIME ZONE NEW.timezone_name;
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER ts_recompute_starts_at_utc
BEFORE INSERT OR UPDATE ON appointments
FOR EACH ROW
EXECUTE FUNCTION recompute_appointment_utc();

Timezone changes with dual columns

If tzdata updates change the rules for a timezone, the derived starts_at_utc values in your database become stale and need to be recomputed. You can do this with a simple UPDATE statement that re-applies the conversion logic:

UPDATE appointments
SET starts_at_utc = local_time AT TIME ZONE timezone_name
WHERE timezone_name = 'America/Vancouver'
  AND starts_at_utc > now();

What about RFC 9557?

In 2024, RFC 9557 was released as a new timestamp formatting that looks like 1996-12-19T16:39:57-08:00[America/Los_Angeles]. A short discussion was had on the pgsql-general forum in November 2025. Usage has not moved forward, as the standard is still quite new, and folks are waiting to see how it gets adopted.

However, the RFC 9557 explicitly stated it was not meant to solve:

future time given as a local time in some specified time zone, where changes to the definition of that time zone (such as a political decision to enact or rescind daylight saving time) affect the instant in time represented by the timestamp;

So, stick with dual column pattern for IRL times sufficiently in the future.

What to do if tzdata has already updated?

If you have already updated tzdata package for the new time zones, and your column values are assigned unknown UTC shifts, and your database records future times for entities in British Columbia, you've got a data project on your hands. Ideally, you would:

Find or estimate when the tzdata package was updated
Find all of the potentially incorrect records
Identify potentially impacted rows using updated_at timestamps after the tzdata update
Make a plan for notifying users of the time-shift adjustment, with potential plan to opt out or opt in
Test time-shift migration against potentially impacted rows on a non-production dataset
Run a backup, then run the time-shift migration on production
Add a UI element for calendar items impacted by the changes
When the now defunct November time change approaches, notify users again of potential timezone issues

Having a population of 5.8M people, British Columbia changing timezone preferences will affect some datasets broadly, and others not at all. Don't get caught by time zone changes; it is surprising how often the tzdata package is updated.

Postgres Serials Should be BIGINT (and How to Migrate)

Elizabeth.Christensen@crunchydata.com (Elizabeth Christensen) — Tue, 20 Jan 2026 08:00:00 EST

Lots of us started with a Postgres database that incremented with an id SERIAL PRIMARY KEY. This was the Postgres standard for many years for data columns that auto incremented. The SERIAL is a shorthand for an integer data type that is automatically incremented. However as your data grows in size, SERIALs and INTs can run the risk of an integer overflow as they get closer to 2 Billion uses.

We covered a lot of this in a blog post The Integer at the End of the Universe: Integer Overflow in Postgres a few years ago. Since that was published we’ve helped a number of customers with this problem and I wanted to refresh the ideas and include some troubleshooting steps that can be helpful. I also think that BIGINT is more cost effective than folks realize.

SERIAL and BIGSERIAL are just shorthands and map directly to the INT and BIGINT data types. While something like CREATE TABLE user_events (id SERIAL PRIMARY KEY) would have been common in the past, the best practice now is BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY is recommended. SERIAL/ BIGSERIAL are not SQL standard and the GENERATED ALWAYS keyword prevents accidental inserts, guaranteeing the database manages the sequence instead of a manual or application based addition.

INT - goes up to 2.1 Billion (2,147,483,647) and more if you do negative numbers. INT takes up 4 bytes per row column.
BIGINT- goes up 9.22 quintillion (9,223,372,036,854,775,807) and needs a 8-bytes for storage.

Serials vs UUID

Before I continue talking about serials in Postgres, it is worth noting that Postgres also has robust UUID support, including v7 which was just released. If you decide to go with UUID, great. This makes a ton of sense for things that can be URLs or are across systems. However not all ids need to be UUIDs, so lots of folks still continue with a serialized / incremented integers.

Cost difference between INT and BIGINT

Postgres does not pack data tightly like a text file. It writes data in aligned tuples / rows, and standard 64-bit servers require data to line up on 8-byte boundaries. In many table layouts, INT and BIGINT consume the exact same amount of disk space. The "savings" of INT are often eaten by empty padding bytes.

Think of this sample table:

INT

Header: 24 bytes (Standard row overhead)
Data: 4 bytes (INT)
Padding: PostgreSQL adds 4 empty bytes to fill the gap so the next row starts on an 8-byte boundary.
Total per Row: $24 + 4 + 4 = 32

BIGINT

Header: 24 bytes (Standard row overhead)
Data: 8 bytes (BIGINT)
Padding: 0 bytes (Already perfectly aligned to 8 bytes).
Total per Row: $24 + 8 + 0 = 32

You pay $0.00 extra for using BIGINT.

Even in the scenario where your specific column order does result in a true 4-byte increase per row for BIGINT, the costs are negligible. Let’s say you have 4 extra bytes per row for a billion rows, that’s just ~4 GB. On Crunchy Bridge that’s about .40 cents a month (similar on other modern clouds).

Using BIGINT instead of INT for a database bound for production sequencing is probably the safer bet if you’re logging anything like timestamps, page hits, or things that will be incrementing to the millions or billions. Avoiding the man hours and cost to do an in-place data type change of this nature is worth it.

Live data type change in Postgres - the atomic swap

Ok, let’s say I’ve convinced you to move to BIGINT now. Maybe you’re close to integer wraparound or maybe you’re small enough that you can do this now before it becomes a bigger headache.

Changing a production data column type is always tricky business. The data type change needs to be done across millions and billions of rows in production, but:

We can’t lock the table
We don’t want to take downtime
We need to preserve the current increments

Luckily our support team helps folks often with these types of changes and with this blog I’ve collected notes and helpful tips over dozens of these projects for this blog post.

The foundational strategy for this migration is to perform the bulk of the work asynchronously—while the application remains online—by creating a new BIGINT column, backfilling the data, and then performing a quick, single-transaction switchover. We like to call this changeover an atomic swap. Atomic swap is a specific technique used to switch a live table with a new version of itself without taking the application offline

Here is the high-level plan:

Add a new BIGINT column, sequence, and a unique index. Backfill the old id values into the new column in batches.
Changeover (Brief Downtime): Lock the table, complete the final backfill, drop old constraints, rename columns (id to id_old, id_new to id), and add a non-validated NOT NULL constraint.
Validate the NOT NULL constraint, promote the column to a Primary Key, and clean up.

Set Up the Test Environment

I’ll provide some sample code for doing a full INT to BIGINT changever. This will make more sense with a sample table that mimics a real-world scenario where the SERIAL primary key is the bottleneck. I’ve also added steps for a foreign key constraint because we see this frequently.

-- 1. Create the Parent Table (Standard SERIAL / INT)
CREATE TABLE user_events (
    id SERIAL PRIMARY KEY,
    data TEXT,
    created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT now()
);

-- 2. Create a Child Table (Foreign Key Dependency)
CREATE TABLE user_events_log (
    log_id SERIAL PRIMARY KEY,
    event_id INTEGER NOT NULL,
    log_message TEXT,
    created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT now(),
    CONSTRAINT fk_user_events
        FOREIGN KEY (event_id)
        REFERENCES user_events (id)
);

-- 3. Populate Initial Data (100k rows)
INSERT INTO user_events (data, created_at)
SELECT 'Historical Data', NOW() - (random() * (interval '90 days'))
FROM generate_series(1, 100000);

INSERT INTO user_events_log (event_id, log_message)
SELECT id, 'Log entry for event ' || id
FROM user_events;

-- We start inserting rows in the background to prove the migration is "Online".  (You may need to configure pg_cron in your environment for this to work.)
CREATE EXTENSION IF NOT EXISTS pg_cron;

SELECT cron.schedule(
    'generate-events-traffic',
    '2 seconds', -- Runs every 2 seconds
    $$
    INSERT INTO user_events (data, created_at)
    SELECT 'Live Incoming Traffic', NOW()
    FROM generate_series(1, 1000);
    $$
);

Add the New BIGINT Columns

We add the column allowing NULLs. Later when we create the primary key index, NULLs will not be allowed. This is a quick metadata change even to a large table. It does take a short lock on the table, but only for a tiny blip because we’re creating a new column.

ALTER TABLE user_events ADD COLUMN id_new BIGINT;
ALTER TABLE user_events_log ADD COLUMN event_id_new BIGINT;

If you’re doing the full test, a trigger ensures any new rows inserted into this table will get their 'id_new' field populated automatically.

CREATE OR REPLACE FUNCTION sync_id_new()
RETURNS TRIGGER AS $$
BEGIN
    NEW.id_new := NEW.id;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_sync_id_new
BEFORE INSERT ON user_events
FOR EACH ROW
EXECUTE FUNCTION sync_id_new();

Backfill in batches

Now we can backfill the new column from the old one. We’ll do this in batches to avoid a massive transaction that could cause replication lag or I/O spikes. We write this as a PROCEDURE to allow us to specify the batch size and sleep time, also allowing it to COMMIT between batches.

CREATE OR REPLACE PROCEDURE backfill_id_new(batch_size INTEGER, sleep_time FLOAT)
AS $$
DECLARE
    rows_updated BIGINT := 0;
    max_id_to_process BIGINT;
BEGIN
    -- We define a "high water mark" so we don't chase the moving target forever.
    -- We know any rows higher than this value will not need to be backfilled.

    SELECT MAX(id) INTO max_id_to_process FROM user_events;

    LOOP
        WITH rows_to_update AS (
            SELECT id
            FROM user_events
            WHERE id_new IS NULL
            AND id <= max_id_to_process
            LIMIT batch_size
            FOR UPDATE SKIP LOCKED
        )
        UPDATE user_events m
        SET id_new = r.id
        FROM rows_to_update r
        WHERE m.id = r.id;

        GET DIAGNOSTICS rows_updated = ROW_COUNT;

        COMMIT;

        EXIT WHEN rows_updated = 0;

        PERFORM pg_sleep(sleep_time);
    END LOOP;
END;
$$ LANGUAGE plpgsql;

-- Run Backfill on the main table
CALL backfill_id_new(1000, 0.5);

-- Backfill Child Table (We are using a simple update for our test script, in
-- practice you would use the same batching approach in prod)
UPDATE user_events_log SET event_id_new = event_id;

Batch vacuum

For a small test like this one, you don’t need to vacuum but as we’ve found with larger production moves, regularly running VACUUM is crucial during the backfill process to clean up dead rows created by the UPDATE statements. The old rows need to be cleaned up since we have new rows that have both INT and BIGINT and this cleanup prevents table bloat.

-- Run this command after every 5-10 backfill batches (e.g., every 500,000 rows)
VACUUM (ANALYZE, VERBOSE) user_events;
VACUUM (ANALYZE, VERBOSE) user_events_log;

You may need to play around with your batch size. Instead of using very large batch sizes (e.g., 4 million rows), stick to a smaller, efficient size (like 100,000 rows). The overall time for a smaller batch plus a vacuum proved to be more efficient than a single massive batch followed by a prolonged vacuum.

Create an index concurrently

We can now create the necessary unique index, which will eventually enforce the primary key constraint. Using CONCURRENTLY is the key to maintaining uptime.

-- 1. Ensure all backfilled rows are NOT NULL for the index creation
ALTER TABLE user_events
ALTER COLUMN id_new SET NOT NULL;

-- 2. Create the unique index CONCURRENTLY (non-locking DML)
CREATE UNIQUE INDEX CONCURRENTLY user_events_id_new_idx ON user_events (id_new);

Final catch-up and sequence configuration

Before the final swap, we perform a quick update on any rows inserted since the initial backfill and configure the sequence to start from the highest existing ID.

-- 1. Catch-up: Update any rows that were inserted during the batch backfill
-- This should be fast, as it only targets newly inserted rows (id_new IS NULL)
UPDATE user_events
SET id_new = id
WHERE id_new IS NULL;

-- 2. Get the new sequence ready to continue from the largest existing ID
-- SERIAL uses an underlying sequence. We rename it to use for IDENTITY
ALTER SEQUENCE user_events_id_seq RENAME TO user_events_id_identity_seq;

-- Set the sequence to the current max value of the old ID (plus a buffer, e.g., 1000)
SELECT setval('user_events_id_identity_seq', (SELECT MAX(id) FROM user_events) + 1000, false);

Updating foreign key columns on the child table

Any table that has a foreign key referencing the primary table's ID column must be updated to BIGINT before the main table's switchover is completed. 🫠

This process is simpler as these columns are not primary keys, but it still requires a process of adding a new BIGINT foreign key column, backfilling, and performing a quick rename switchover for each referencing table. The trick here is adding the constraint as NOT VALID and making it valid later.

-- 1. Enforce NOT NULL on Parent
ALTER TABLE user_events ALTER COLUMN id_new SET NOT NULL;

-- 2. Create Unique Index Concurrently, this prepares the future Primary Key without locking writes)
CREATE UNIQUE INDEX CONCURRENTLY user_events_id_new_idx ON user_events (id_new);

-- 3. Add Foreign Key Constraint to Child (NOT VALID)
ALTER TABLE user_events_log
    ADD CONSTRAINT fk_user_events_new
    FOREIGN KEY (event_id_new)
    REFERENCES user_events (id_new)
    NOT VALID;

-- 4. Validate FK (Scans table, but does not block parent updates)
ALTER TABLE user_events_log VALIDATE CONSTRAINT fk_user_events_new;

-- Done before the parent swap. Brief exclusive lock on child table only.

BEGIN;
    LOCK TABLE user_events_log IN ACCESS EXCLUSIVE MODE;

    -- Drop old FK and column
    ALTER TABLE user_events_log DROP CONSTRAINT fk_user_events;
    ALTER TABLE user_events_log DROP COLUMN event_id;

    -- Rename new column/constraint to match old names
    ALTER TABLE user_events_log RENAME COLUMN event_id_new TO event_id;
    ALTER TABLE user_events_log RENAME CONSTRAINT fk_user_events_new TO fk_user_events;
COMMIT;

The atomic swap (brief lock)

If you followed along for the sake of testing, stop your cron job SELECT cron.unschedule('generate-events-traffic');.

This is the final step, done inside a single transaction. It requires an exclusive lock, but since the index is already built, this step is purely metadata and should take milliseconds.

BEGIN;
    LOCK TABLE user_events IN ACCESS EXCLUSIVE MODE;

    -- Drop the Sync Trigger (We don't need it after the swap)
    DROP TRIGGER trg_sync_id_new ON user_events;
    DROP FUNCTION sync_id_new;

    -- Drop old PK constraint
    ALTER TABLE user_events DROP CONSTRAINT user_events_pkey;

    -- Make the new column active, drop old one
    ALTER TABLE user_events DROP COLUMN id;
    ALTER TABLE user_events RENAME COLUMN id_new TO id;

    -- Add IDENTITY (Creates a fresh sequence automatically)
    ALTER TABLE user_events
    ALTER COLUMN id ADD GENERATED ALWAYS AS IDENTITY;

    -- Sync the new Sequence to the Data
    SELECT setval(pg_get_serial_sequence('user_events', 'id'), (SELECT MAX(id) FROM user_events));

    -- Re-add Primary Key (Using the pre-built index)
    -- Postgres will automatically rename the index 'user_events_id_new_idx' to 'user_events_pkey'
    ALTER TABLE user_events
    ADD CONSTRAINT user_events_pkey PRIMARY KEY USING INDEX user_events_id_new_idx;

COMMIT;

Conclusion

BIGINT is cheap! You might want to do a migration soon.

Migrating a sequencing column from INT to BIGINT is a complex database refactoring project, but by utilizing Postgres features like unique indexes, sequences, and the NOT VALID constraint trick, it can be executed with minimal application downtime.

Key Takeaways:

Do as much work as possible (adding new column, index, backfilling) while the application is online.
Test batch sizes and vacuum to get to a backfill process that is efficient
Update referencing foreign key columns to BIGINT before the main table switch.
Atomic switchover: Execute the column rename and constraint setup in a single, quick transaction.

As always, test the entire process on a non-production fork and ensure the plan works as expected before committing to production.

How to Read Postgres EXPLAIN: A Guide to Scan Types

Elizabeth.Christensen@crunchydata.com (Elizabeth Christensen) — Thu, 04 Dec 2025 08:00:00 EST

The secret to unlocking performance gains often lies not just in what you ask in a query, but in how Postgres finds the answer. The Postgres EXPLAIN system is great for understanding how data is being queried. One of secretes to reading EXPLAIN plans is understanding the type of scan done to retrieve the data. The scan type can be the difference between a lightning-fast response or a slow query.

Today I’ll break down the most common scan types, how they work, and when you’ll see them in your queries.

Sequential scan

This type of data scan reads the entire table, row by row checking to see what matches the query conditions. If you have a WHERE or FILTER, Postgres just scans each row looking for matches.

Sequence scans are kind of the foundation of how scans are done and for many searches, this is what Postgres will use. For very large data sets, or those queried often, sequential scans are not ideal and an index scan may be faster. For that reason - knowing how to spot a seq scan vs index scan when reading an EXPLAIN plan is one the most important parts of reading a scan type in a query plan.

EXPLAIN select * from accounts;

QUERY PLAN
-------------------------------------------------------------
Seq Scan on accounts  (cost=0.00..22.70 rows=1270 width=36)
(1 row)

Index Scan

When you create an index in Postgres, you’re creating a column or multi-column reference that is stored on disk. Postgres is able to use this index as a map to the data stored in the table. A basic index scan uses a B-tree to quickly find the exact location of the data using a a two-step process: first Postgres finds the entry in the index, uses the reference, and then it fetches the rest of the row data from the table.

EXPLAIN select * from accounts where id = '5';

                                  QUERY PLAN
-------------------------------------------------------------------------------
 Index Scan using accounts_pkey on accounts  (cost=0.15..2.37 rows=1 width=36)
   Index Cond: (id = 5)
(2 rows)

Note that primary keys are automatically indexed with a b-tree index, so queries that involve a primary key may use an index scan.

An index scan is typically faster than a sequential scan in Postgres when a query needs to retrieve only a very small fraction of rows from a large table. Using the index is faster than scanning the whole table.

However, index scans are not always faster. In many situations, Postgres’ query planner will correctly choose a sequential scan. This is typically for cases when the table being scanned is small or the percentage of rows returned outweighs using an index. If a query returns ~10%, a sequential scan is probably faster.

Bitmap Index Scan

If an index scan or a seq scan aren’t the perfect option, Postgres can use the the bitmap index scan as a kind of hybrid approach. It is typically chosen when a query matches too many rows for an regular index scan, but not so many that a sequential scan would be the best option.

This shows up in an EXPLAIN plan as a two-phased approach.

Bitmap Index Scan: First, Postgres scans one or more indexes to create an in-memory "bitmap", a simple map of all the table pages that might contain rows you need.
Bitmap Heap Scan: The bitmap is used to visit the main table. The key here is that it reads the required pages from the disk sequentially, which can be much faster than the random jumping of a standard index scan.

Bitmap index scans are common when a query has multiple filter conditions that each have a separate index. The bitmap scan allows the database to use separate indexes on different columns simultaneously. You’ll see this scan come up with WHERE conditions joined by AND or OR operators.

EXPLAIN SELECT customer_id, registration_date
FROM customer_records
WHERE gender = 'F'
  AND state_code = 'KS';
                                                               QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on customer_records  (cost=835.78..8669.29 rows=49226 width=12) (actual time=5.717..38.642 rows=50184.00 loops=1)
   Recheck Cond: (state_code = 'NY'::bpchar)
   Filter: (gender = 'F'::bpchar)
   Rows Removed by Filter: 49682
   Heap Blocks: exact=6370
   Buffers: shared hit=6370 read=87
   ->  Bitmap Index Scan on idx_customer_state  (cost=0.00..823.48 rows=97567 width=0) (actual time=4.377..4.378 rows=99866.00 loops=1)
         Index Cond: (state_code = 'NY'::bpchar)
         Index Searches: 1
         Buffers: shared read=87
 Planning:
   Buffers: shared hit=27 read=2
 Planning Time: 0.774 ms
 Execution Time: 40.572 ms
(14 rows)

Parallel Sequential Scan

You will see a parallel sequential scan when Postgres uses multiple background workers to perform more than one sequential scan on a single large table at the same time. The table is broken into chunks, and each worker gets a chunk to scan, and the results are combined at the end in a gather process. Depending on your query - you may also have an aggregate or sort after the parallel queries and before the final gather. This is part of Postgres’ parallel query function.

EXPLAIN (ANALYZE, VERBOSE, BUFFERS)
SELECT id, data_value
FROM parallel_test
WHERE data_value < 100000
ORDER BY data_value DESC
LIMIT 1000;

                                                                         QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=161310.11..161431.04 rows=1000 width=16) (actual time=130.300..140.555 rows=1000.00 loops=1)
   Output: id, data_value
   Buffers: shared hit=142685
   ->  Gather Merge  (cost=161310.11..220311.14 rows=487915 width=16) (actual time=130.299..140.468 rows=1000.00 loops=1)
         Output: id, data_value
         Workers Planned: 5
         Workers Launched: 5
         Buffers: shared hit=142685
         ->  Sort  (cost=160310.04..160553.99 rows=97583 width=16) (actual time=112.942..112.973 rows=861.17 loops=6)
               Output: id, data_value
               Sort Key: parallel_test.data_value DESC
               Sort Method: top-N heapsort  Memory: 163kB
               Buffers: shared hit=142685
               Worker 0:  actual time=112.535..112.571 rows=1000.00 loops=1
                 Sort Method: top-N heapsort  Memory: 164kB
                 Buffers: shared hit=21729
               Worker 1:  actual time=112.271..112.308 rows=1000.00 loops=1
                 Sort Method: top-N heapsort  Memory: 164kB
                 Buffers: shared hit=21573
               Worker 2:  actual time=112.465..112.500 rows=1000.00 loops=1
                 Sort Method: top-N heapsort  Memory: 164kB
                 Buffers: shared hit=20549
               Worker 3:  actual time=99.099..99.133 rows=1000.00 loops=1
                 Sort Method: top-N heapsort  Memory: 163kB
                 Buffers: shared hit=17033
               Worker 4:  actual time=112.333..112.368 rows=1000.00 loops=1
                 Sort Method: top-N heapsort  Memory: 163kB
                 Buffers: shared hit=19964
               ->  Parallel Seq Scan on public.parallel_test  (cost=0.00..154959.67 rows=97583 width=16) (actual time=19.238..99.868 rows=83250.83 loops=6)
                     Output: id, data_value
                     Filter: (parallel_test.data_value < '100000'::numeric)
                     Rows Removed by Filter: 750082
                     Buffers: shared hit=142500
                     Worker 0:  actual time=18.837..99.169 rows=83026.00 loops=1
                       Buffers: shared hit=21692
                     Worker 1:  actual time=18.594..99.301 rows=84378.00 loops=1
                       Buffers: shared hit=21536
                     Worker 2:  actual time=18.706..99.551 rows=79196.00 loops=1
                       Buffers: shared hit=20512
                     Worker 3:  actual time=5.308..86.023 rows=81187.00 loops=1
                       Buffers: shared hit=16996
                     Worker 4:  actual time=18.694..99.497 rows=83574.00 loops=1
                       Buffers: shared hit=19927
 Planning:
   Buffers: shared hit=15
 Planning Time: 0.315 ms
 Execution Time: 140.635 ms
(47 rows)

Parallel index scan

A parallel index scan uses the same parallel workers to scan through an index concurrently. This uses the same methodology of the index scan - except that multiple workers are doing it simultaneously. Each process reads a different part of the index and returns results. Like the other parallel scans, this ends in a gather.

You will see a parallel index scan done when the indexes and tables involved are very large - and the overall operation to split things up and gather them at the end is faster than handing the job to a single worker.

EXPLAIN (ANALYZE, VERBOSE, BUFFERS)
SELECT data_id, filler_text
FROM parallel_index_test
WHERE data_id BETWEEN 1000000 AND 2000000;

                                                                                QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Gather  (cost=0.43..34560.34 rows=995971 width=109) (actual time=1.014..145.796 rows=1000001.00 loops=1)
   Output: data_id, filler_text
   Workers Planned: 4
   Workers Launched: 4
   Buffers: shared hit=23385
   ->  Parallel Index Scan using idx_data_id on public.parallel_index_test  (cost=0.43..33564.37 rows=248993 width=109) (actual time=0.941..38.211 rows=200000.20 loops=5)
         Output: data_id, filler_text
         Index Cond: ((parallel_index_test.data_id >= 1000000) AND (parallel_index_test.data_id <= 2000000))
         Index Searches: 1
         Buffers: shared hit=23385
         Worker 0:  actual time=2.104..45.540 rows=240638.00 loops=1
           Buffers: shared hit=5640
         Worker 1:  actual time=2.174..45.169 rows=240096.00 loops=1
           Buffers: shared hit=5638
         Worker 2:  actual time=0.067..45.380 rows=242658.00 loops=1
           Buffers: shared hit=5693
         Worker 3:  actual time=0.306..45.122 rows=242292.00 loops=1
           Buffers: shared hit=5686
 Planning:
   Buffers: shared hit=4
 Planning Time: 0.526 ms
 Execution Time: 180.660 ms
(22 rows)

Index-Only Scan

An Index-Only Scan is the superstar of scans and answers the entire query using only the information stored within the index itself. Index only scans are also called “covering indexes” meaning the index itself covers all the data. It never even has to touch the main table. Index only scans are a huge performance win because they’re very fast - no information needs to be retrieved from the heap table. They also typically use less i/o resources because indexes are very cache friendly and often in shared buffers - meaning no data needs to be read for the underlying disk.

Queries benefit from a covering index in these situations:

The query is very frequently executed.
The current query is performing a standard index scan followed by many slow disk reads (heap fetches) and using i/o.
The query only requires a small subset of the table's columns, for example you select only three columns from a table of twenty.
The columns have a low write frequency. Any column that is indexed must be written to disk and the index, so if you start adding covering indexes for all your columns - you’re essentially creating write amplification.
The new index, which must cover all needed columns, won't be excessively large. Indexes are stored on disk so you don’t want to cause storage issues.

EXPLAIN (ANALYZE, VERBOSE, BUFFERS)
SELECT code, status
FROM index_only_test
WHERE code > 'CODE_050000'
ORDER BY code
LIMIT 100;
                                                                           QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.42..2.60 rows=100 width=13) (actual time=0.346..0.362 rows=100.00 loops=1)
   Output: code, status
   Buffers: shared hit=1 read=3
   ->  Index Only Scan using idx_code_status on public.index_only_test  (cost=0.42..1068.02 rows=49000 width=13) (actual time=0.345..0.352 rows=100.00 loops=1)
         Output: code, status
         Index Cond: (index_only_test.code > 'CODE_050000'::text)
         Heap Fetches: 0
         Index Searches: 1
         Buffers: shared hit=1 read=3
 Planning:
   Buffers: shared hit=19
 Planning Time: 1.838 ms
 Execution Time: 0.385 ms
(13 rows)

Summary

We’ve covered all the major scan types so now reading your EXPLAIN plans will be a little easier.

Seq scan - Postgres looks through the whole table in sequential order to find the query data
Index scan - Postgres first looks at the index and then fetches the row data the index pointed to
Bitmap index scan - Postgres first read the index and created a bitmap list matching rows. Second, Postgres read the data heap using the bitmap in a more efficient method than a sequential scan.
Parallel scan - Postgres used multiple parallel workers to scan the table and data was gathered at the end
Parallel index scan - Postgres used multiple workers to do an index scan and data was gathered at the end
Index only scan- All data for the query was in the index

And here’s everything all in one graphic: