We covered a lot of this in a blog post The Integer at the End of the Universe: Integer Overflow in Postgres a few years ago. Since that was published we’ve helped a number of customers with this problem and I wanted to refresh the ideas and include some troubleshooting steps that can be helpful. I also think that BIGINT is more cost effective than folks realize.

SERIAL and BIGSERIAL are just shorthands and map directly to the INT and BIGINT data types. While something like CREATE TABLE user_events (id SERIAL PRIMARY KEY) would have been common in the past, the best practice now is BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY is recommended. SERIAL/ BIGSERIAL are not SQL standard and the GENERATED ALWAYS keyword prevents accidental inserts, guaranteeing the database manages the sequence instead of a manual or application based addition.

• INT - goes up to 2.1 Billion (2,147,483,647) and more if you do negative numbers. INT takes up 4 bytes per row column.
• BIGINT- goes up 9.22 quintillion (9,223,372,036,854,775,807) and needs a 8-bytes for storage.

Serials vs UUID

Before I continue talking about serials in Postgres, it is worth noting that Postgres also has robust UUID support, including v7 which was just released. If you decide to go with UUID, great. This makes a ton of sense for things that can be URLs or are across systems. However not all ids need to be UUIDs, so lots of folks still continue with a serialized / incremented integers.

Cost difference between INT and BIGINT

Postgres does not pack data tightly like a text file. It writes data in aligned tuples /  rows, and standard 64-bit servers require data to line up on 8-byte boundaries. In many table layouts, INT and BIGINT consume the exact same amount of disk space. The "savings" of INT are often eaten by empty padding bytes.

Think of this sample table:

INT

• Header: 24 bytes (Standard row overhead)
• Data: 4 bytes (INT)
• Padding: PostgreSQL adds 4 empty bytes to fill the gap so the next row starts on an 8-byte boundary.
• Total per Row: $24 + 4 + 4 = 32

BIGINT

• Header: 24 bytes (Standard row overhead)
• Data: 8 bytes (BIGINT)
• Padding: 0 bytes (Already perfectly aligned to 8 bytes).
• Total per Row: $24 + 8 + 0 = 32

You pay $0.00 extra for using BIGINT.

Even in the scenario where your specific column order does result in a true 4-byte increase per row for BIGINT, the costs are negligible. Let’s say you have 4 extra bytes per row for a billion rows, that’s just ~4 GB. On Crunchy Bridge that’s about .40 cents a month (similar on other modern clouds).

Using BIGINT instead of INT for a database bound for production sequencing is probably the safer bet if you’re logging anything like timestamps, page hits, or things that will be incrementing to the millions or billions. Avoiding the man hours and cost to do an in-place data type change of this nature is worth it.

Live data type change in Postgres - the atomic swap

Ok, let’s say I’ve convinced you to move to BIGINT now. Maybe you’re close to integer wraparound or maybe you’re small enough that you can do this now before it becomes a bigger headache.

Changing a production data column type is always tricky business. The data type change needs to be done across millions and billions of rows in production, but:

• We can’t lock the table
• We don’t want to take downtime
• We need to preserve the current increments

Luckily our support team helps folks often with these types of changes and with this blog I’ve collected notes and helpful tips over dozens of these projects for this blog post.

The foundational strategy for this migration is to perform the bulk of the work asynchronously—while the application remains online—by creating a new BIGINT column, backfilling the data, and then performing a quick, single-transaction switchover. We like to call this changeover an atomic swap. Atomic swap is a specific technique used to switch a live table with a new version of itself without taking the application offline

Here is the high-level plan:

1. Add a new BIGINT column, sequence, and a unique index. Backfill the old id values into the new column in batches.
2. Changeover (Brief Downtime): Lock the table, complete the final backfill, drop old constraints, rename columns (id to id_old, id_new to id), and add a non-validated NOT NULL constraint.
3. Validate the NOT NULL constraint, promote the column to a Primary Key, and clean up.

Set Up the Test Environment

I’ll provide some sample code for doing a full INT to BIGINT changever. This will make more sense with a sample table that mimics a real-world scenario where the SERIAL primary key is the bottleneck. I’ve also added steps for a foreign key constraint because we see this frequently.

-- 1. Create the Parent Table (Standard SERIAL / INT)
CREATE TABLE user_events (
    id SERIAL PRIMARY KEY,
    data TEXT,
    created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT now()
);

-- 2. Create a Child Table (Foreign Key Dependency)
CREATE TABLE user_events_log (
    log_id SERIAL PRIMARY KEY,
    event_id INTEGER NOT NULL,
    log_message TEXT,
    created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT now(),
    CONSTRAINT fk_user_events
        FOREIGN KEY (event_id)
        REFERENCES user_events (id)
);

-- 3. Populate Initial Data (100k rows)
INSERT INTO user_events (data, created_at)
SELECT 'Historical Data', NOW() - (random() * (interval '90 days'))
FROM generate_series(1, 100000);

INSERT INTO user_events_log (event_id, log_message)
SELECT id, 'Log entry for event ' || id
FROM user_events;

-- We start inserting rows in the background to prove the migration is "Online".  (You may need to configure pg_cron in your environment for this to work.)
CREATE EXTENSION IF NOT EXISTS pg_cron;

SELECT cron.schedule(
    'generate-events-traffic',
    '2 seconds', -- Runs every 2 seconds
    $$
    INSERT INTO user_events (data, created_at)
    SELECT 'Live Incoming Traffic', NOW()
    FROM generate_series(1, 1000);
    $$
);

Add the New BIGINT Columns

We add the column allowing NULLs. Later when we create the primary key index, NULLs will not be allowed. This is a quick metadata change even to a large table. It does take a short lock on the table, but only for a tiny blip because we’re creating a new column.

ALTER TABLE user_events ADD COLUMN id_new BIGINT;
ALTER TABLE user_events_log ADD COLUMN event_id_new BIGINT;

If you’re doing the full test, a trigger ensures any new rows inserted into this table will get their 'id_new' field populated automatically.

CREATE OR REPLACE FUNCTION sync_id_new()
RETURNS TRIGGER AS $$
BEGIN
    NEW.id_new := NEW.id;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_sync_id_new
BEFORE INSERT ON user_events
FOR EACH ROW
EXECUTE FUNCTION sync_id_new();

Backfill in batches

Now we can backfill the new column from the old one. We’ll do this in batches to avoid a massive transaction that could cause replication lag or I/O spikes.  We write this as a PROCEDURE to allow us to specify the batch size and sleep time, also allowing it to COMMIT between batches.

CREATE OR REPLACE PROCEDURE backfill_id_new(batch_size INTEGER, sleep_time FLOAT)
AS $$
DECLARE
    rows_updated BIGINT := 0;
    max_id_to_process BIGINT;
BEGIN
    -- We define a "high water mark" so we don't chase the moving target forever.
    -- We know any rows higher than this value will not need to be backfilled.

    SELECT MAX(id) INTO max_id_to_process FROM user_events;

    LOOP
        WITH rows_to_update AS (
            SELECT id
            FROM user_events
            WHERE id_new IS NULL
            AND id <= max_id_to_process
            LIMIT batch_size
            FOR UPDATE SKIP LOCKED
        )
        UPDATE user_events m
        SET id_new = r.id
        FROM rows_to_update r
        WHERE m.id = r.id;

        GET DIAGNOSTICS rows_updated = ROW_COUNT;

        COMMIT;

        EXIT WHEN rows_updated = 0;

        PERFORM pg_sleep(sleep_time);
    END LOOP;
END;
$$ LANGUAGE plpgsql;

-- Run Backfill on the main table
CALL backfill_id_new(1000, 0.5);

-- Backfill Child Table (We are using a simple update for our test script, in
-- practice you would use the same batching approach in prod)
UPDATE user_events_log SET event_id_new = event_id;

Batch vacuum

For a small test like this one, you don’t need to vacuum but as we’ve found with larger production moves, regularly running VACUUM is crucial during the backfill process to clean up dead rows created by the UPDATE statements. The old rows need to be cleaned up since we have new rows that have both INT and BIGINT and this cleanup prevents table bloat.

-- Run this command after every 5-10 backfill batches (e.g., every 500,000 rows)
VACUUM (ANALYZE, VERBOSE) user_events;
VACUUM (ANALYZE, VERBOSE) user_events_log;

You may need to play around with your batch size. Instead of using very large batch sizes (e.g., 4 million rows), stick to a smaller, efficient size (like 100,000 rows). The overall time for a smaller batch plus a vacuum proved to be more efficient than a single massive batch followed by a prolonged vacuum.

Create an index concurrently

We can now create the necessary unique index, which will eventually enforce the primary key constraint. Using CONCURRENTLY is the key to maintaining uptime.

-- 1. Ensure all backfilled rows are NOT NULL for the index creation
ALTER TABLE user_events
ALTER COLUMN id_new SET NOT NULL;

-- 2. Create the unique index CONCURRENTLY (non-locking DML)
CREATE UNIQUE INDEX CONCURRENTLY user_events_id_new_idx ON user_events (id_new);

Final catch-up and sequence configuration

Before the final swap, we perform a quick update on any rows inserted since the initial backfill and configure the sequence to start from the highest existing ID.

-- 1. Catch-up: Update any rows that were inserted during the batch backfill
-- This should be fast, as it only targets newly inserted rows (id_new IS NULL)
UPDATE user_events
SET id_new = id
WHERE id_new IS NULL;

-- 2. Get the new sequence ready to continue from the largest existing ID
-- SERIAL uses an underlying sequence. We rename it to use for IDENTITY
ALTER SEQUENCE user_events_id_seq RENAME TO user_events_id_identity_seq;

-- Set the sequence to the current max value of the old ID (plus a buffer, e.g., 1000)
SELECT setval('user_events_id_identity_seq', (SELECT MAX(id) FROM user_events) + 1000, false);

Updating foreign key columns on the child table

Any table that has a foreign key referencing the primary table's ID column must be updated to BIGINT before the main table's switchover is completed. 🫠

This process is simpler as these columns are not primary keys, but it still requires a process of adding a new BIGINT foreign key column, backfilling, and performing a quick rename switchover for each referencing table. The trick here is adding the constraint as NOT VALID and making it valid later.

-- 1. Enforce NOT NULL on Parent
ALTER TABLE user_events ALTER COLUMN id_new SET NOT NULL;

-- 2. Create Unique Index Concurrently, this prepares the future Primary Key without locking writes)
CREATE UNIQUE INDEX CONCURRENTLY user_events_id_new_idx ON user_events (id_new);

-- 3. Add Foreign Key Constraint to Child (NOT VALID)
ALTER TABLE user_events_log
    ADD CONSTRAINT fk_user_events_new
    FOREIGN KEY (event_id_new)
    REFERENCES user_events (id_new)
    NOT VALID;

-- 4. Validate FK (Scans table, but does not block parent updates)
ALTER TABLE user_events_log VALIDATE CONSTRAINT fk_user_events_new;

-- Done before the parent swap. Brief exclusive lock on child table only.

BEGIN;
    LOCK TABLE user_events_log IN ACCESS EXCLUSIVE MODE;

    -- Drop old FK and column
    ALTER TABLE user_events_log DROP CONSTRAINT fk_user_events;
    ALTER TABLE user_events_log DROP COLUMN event_id;

    -- Rename new column/constraint to match old names
    ALTER TABLE user_events_log RENAME COLUMN event_id_new TO event_id;
    ALTER TABLE user_events_log RENAME CONSTRAINT fk_user_events_new TO fk_user_events;
COMMIT;

The atomic swap (brief lock)

If you followed along for the sake of testing, stop your cron job SELECT cron.unschedule('generate-events-traffic');.

This is the final step, done inside a single transaction. It requires an exclusive lock, but since the index is already built, this step is purely metadata and should take milliseconds.

BEGIN;
    LOCK TABLE user_events IN ACCESS EXCLUSIVE MODE;

    -- Drop the Sync Trigger (We don't need it after the swap)
    DROP TRIGGER trg_sync_id_new ON user_events;
    DROP FUNCTION sync_id_new;

    -- Drop old PK constraint
    ALTER TABLE user_events DROP CONSTRAINT user_events_pkey;

    -- Make the new column active, drop old one
    ALTER TABLE user_events DROP COLUMN id;
    ALTER TABLE user_events RENAME COLUMN id_new TO id;

    -- Add IDENTITY (Creates a fresh sequence automatically)
    ALTER TABLE user_events
    ALTER COLUMN id ADD GENERATED ALWAYS AS IDENTITY;

    -- Sync the new Sequence to the Data
    SELECT setval(pg_get_serial_sequence('user_events', 'id'), (SELECT MAX(id) FROM user_events));

    -- Re-add Primary Key (Using the pre-built index)
    -- Postgres will automatically rename the index 'user_events_id_new_idx' to 'user_events_pkey'
    ALTER TABLE user_events
    ADD CONSTRAINT user_events_pkey PRIMARY KEY USING INDEX user_events_id_new_idx;

COMMIT;

Conclusion

BIGINT is cheap! You might want to do a migration soon.

Migrating a sequencing column from INT to BIGINT is a complex database refactoring project, but by utilizing Postgres features like unique indexes, sequences, and the NOT VALID constraint trick, it can be executed with minimal application downtime.

Key Takeaways:

• Do as much work as possible (adding new column, index, backfilling) while the application is online.
• Test batch sizes and vacuum to get to a backfill process that is efficient
• Update referencing foreign key columns to BIGINT before the main table switch.
• Atomic switchover: Execute the column rename and constraint setup in a single, quick transaction.

As always, test the entire process on a non-production fork and ensure the plan works as expected before committing to production. ]]> 451deab9e40fcfdba428e5a2ab9dde922c5a819d70e38f6ee9224c974df80c3f Tue, 20 Jan 2026 08:00:00 EST 2026-01-20T13:00:00.000Z 2026-01-20T13:00:00.000Z https://www.crunchydata.com/blog/postgres-scan-types-in-explain-plans The secret to unlocking performance gains often lies not just in what you ask in a query, but in how Postgres finds the answer. The Postgres EXPLAIN system is great for understanding how data is being queried. One of secretes to reading EXPLAIN plans is understanding the type of scan done to retrieve the data. The scan type can be the difference between a lightning-fast response or a slow query.

Today I’ll break down the most common scan types, how they work, and when you’ll see them in your queries.

Sequential scan

This type of data scan reads the entire table, row by row checking to see what matches the query conditions. If you have a WHERE or FILTER, Postgres just scans each row looking for matches.

Sequence scans are kind of the foundation of how scans are done and for many searches, this is what Postgres will use. For very large data sets, or those queried often, sequential scans are not ideal and an index scan may be faster. For that reason - knowing how to spot a seq scan vs index scan when reading an EXPLAIN plan is one the most important parts of reading a scan type in a query plan.

EXPLAIN select * from accounts;

QUERY PLAN
-------------------------------------------------------------
Seq Scan on accounts  (cost=0.00..22.70 rows=1270 width=36)
(1 row)

Index Scan

When you create an index in Postgres, you’re creating a column or multi-column reference that is stored on disk. Postgres is able to use this index as a map to the data stored in the table. A basic index scan uses a B-tree to quickly find the exact location of the data using a a two-step process: first Postgres finds the entry in the index, uses the reference, and then it fetches the rest of the row data from the table.

EXPLAIN select * from accounts where id = '5';

                                  QUERY PLAN
-------------------------------------------------------------------------------
 Index Scan using accounts_pkey on accounts  (cost=0.15..2.37 rows=1 width=36)
   Index Cond: (id = 5)
(2 rows)

Note that primary keys are automatically indexed with a b-tree index, so queries that involve a primary key may use an index scan.

An index scan is typically faster than a sequential scan in Postgres when a query needs to retrieve only a very small fraction of rows from a large table. Using the index is faster than scanning the whole table.

However, index scans are not always faster. In many situations, Postgres’ query planner will correctly choose a sequential scan. This is typically for cases when the table being scanned is small or the percentage of rows returned outweighs using an index. If a query returns ~10%, a sequential scan is probably faster.

Bitmap Index Scan

If an index scan or a seq scan aren’t the perfect option, Postgres can use the the bitmap index scan as a kind of hybrid approach. It is typically chosen when a query matches too many rows for an regular index scan, but not so many that a sequential scan would be the best option.

This shows up in an EXPLAIN plan as a two-phased approach.

1. Bitmap Index Scan: First, Postgres scans one or more indexes to create an in-memory "bitmap", a simple map of all the table pages that might contain rows you need.
2. Bitmap Heap Scan: The bitmap is used to visit the main table. The key here is that it reads the required pages from the disk sequentially, which can be much faster than the random jumping of a standard index scan.

Bitmap index scans are common when a query has multiple filter conditions that each have a separate index. The bitmap scan allows the database to use separate indexes on different columns simultaneously. You’ll see this scan come up with WHERE conditions joined by AND or OR operators.

EXPLAIN SELECT customer_id, registration_date
FROM customer_records
WHERE gender = 'F'
  AND state_code = 'KS';
                                                               QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on customer_records  (cost=835.78..8669.29 rows=49226 width=12) (actual time=5.717..38.642 rows=50184.00 loops=1)
   Recheck Cond: (state_code = 'NY'::bpchar)
   Filter: (gender = 'F'::bpchar)
   Rows Removed by Filter: 49682
   Heap Blocks: exact=6370
   Buffers: shared hit=6370 read=87
   ->  Bitmap Index Scan on idx_customer_state  (cost=0.00..823.48 rows=97567 width=0) (actual time=4.377..4.378 rows=99866.00 loops=1)
         Index Cond: (state_code = 'NY'::bpchar)
         Index Searches: 1
         Buffers: shared read=87
 Planning:
   Buffers: shared hit=27 read=2
 Planning Time: 0.774 ms
 Execution Time: 40.572 ms
(14 rows)

Parallel Sequential Scan

You will see a parallel sequential scan when Postgres uses multiple background workers to perform more than one sequential scan on a single large table at the same time. The table is broken into chunks, and each worker gets a chunk to scan, and the results are combined at the end in a gather process. Depending on your query - you may also have an aggregate or sort after the parallel queries and before the final gather. This is part of Postgres’ parallel query function.

EXPLAIN (ANALYZE, VERBOSE, BUFFERS)
SELECT id, data_value
FROM parallel_test
WHERE data_value < 100000
ORDER BY data_value DESC
LIMIT 1000;

                                                                         QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=161310.11..161431.04 rows=1000 width=16) (actual time=130.300..140.555 rows=1000.00 loops=1)
   Output: id, data_value
   Buffers: shared hit=142685
   ->  Gather Merge  (cost=161310.11..220311.14 rows=487915 width=16) (actual time=130.299..140.468 rows=1000.00 loops=1)
         Output: id, data_value
         Workers Planned: 5
         Workers Launched: 5
         Buffers: shared hit=142685
         ->  Sort  (cost=160310.04..160553.99 rows=97583 width=16) (actual time=112.942..112.973 rows=861.17 loops=6)
               Output: id, data_value
               Sort Key: parallel_test.data_value DESC
               Sort Method: top-N heapsort  Memory: 163kB
               Buffers: shared hit=142685
               Worker 0:  actual time=112.535..112.571 rows=1000.00 loops=1
                 Sort Method: top-N heapsort  Memory: 164kB
                 Buffers: shared hit=21729
               Worker 1:  actual time=112.271..112.308 rows=1000.00 loops=1
                 Sort Method: top-N heapsort  Memory: 164kB
                 Buffers: shared hit=21573
               Worker 2:  actual time=112.465..112.500 rows=1000.00 loops=1
                 Sort Method: top-N heapsort  Memory: 164kB
                 Buffers: shared hit=20549
               Worker 3:  actual time=99.099..99.133 rows=1000.00 loops=1
                 Sort Method: top-N heapsort  Memory: 163kB
                 Buffers: shared hit=17033
               Worker 4:  actual time=112.333..112.368 rows=1000.00 loops=1
                 Sort Method: top-N heapsort  Memory: 163kB
                 Buffers: shared hit=19964
               ->  Parallel Seq Scan on public.parallel_test  (cost=0.00..154959.67 rows=97583 width=16) (actual time=19.238..99.868 rows=83250.83 loops=6)
                     Output: id, data_value
                     Filter: (parallel_test.data_value < '100000'::numeric)
                     Rows Removed by Filter: 750082
                     Buffers: shared hit=142500
                     Worker 0:  actual time=18.837..99.169 rows=83026.00 loops=1
                       Buffers: shared hit=21692
                     Worker 1:  actual time=18.594..99.301 rows=84378.00 loops=1
                       Buffers: shared hit=21536
                     Worker 2:  actual time=18.706..99.551 rows=79196.00 loops=1
                       Buffers: shared hit=20512
                     Worker 3:  actual time=5.308..86.023 rows=81187.00 loops=1
                       Buffers: shared hit=16996
                     Worker 4:  actual time=18.694..99.497 rows=83574.00 loops=1
                       Buffers: shared hit=19927
 Planning:
   Buffers: shared hit=15
 Planning Time: 0.315 ms
 Execution Time: 140.635 ms
(47 rows)

Parallel index scan

A parallel index scan uses the same parallel workers to scan through an index concurrently. This uses the same methodology of the index scan - except that multiple workers are doing it simultaneously. Each process reads a different part of the index and returns results. Like the other parallel scans, this ends in a gather.

You will see a parallel index scan done when the indexes and tables involved are very large - and the overall operation to split things up and gather them at the end is faster than handing the job to a single worker.

EXPLAIN (ANALYZE, VERBOSE, BUFFERS)
SELECT data_id, filler_text
FROM parallel_index_test
WHERE data_id BETWEEN 1000000 AND 2000000;

                                                                                QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Gather  (cost=0.43..34560.34 rows=995971 width=109) (actual time=1.014..145.796 rows=1000001.00 loops=1)
   Output: data_id, filler_text
   Workers Planned: 4
   Workers Launched: 4
   Buffers: shared hit=23385
   ->  Parallel Index Scan using idx_data_id on public.parallel_index_test  (cost=0.43..33564.37 rows=248993 width=109) (actual time=0.941..38.211 rows=200000.20 loops=5)
         Output: data_id, filler_text
         Index Cond: ((parallel_index_test.data_id >= 1000000) AND (parallel_index_test.data_id <= 2000000))
         Index Searches: 1
         Buffers: shared hit=23385
         Worker 0:  actual time=2.104..45.540 rows=240638.00 loops=1
           Buffers: shared hit=5640
         Worker 1:  actual time=2.174..45.169 rows=240096.00 loops=1
           Buffers: shared hit=5638
         Worker 2:  actual time=0.067..45.380 rows=242658.00 loops=1
           Buffers: shared hit=5693
         Worker 3:  actual time=0.306..45.122 rows=242292.00 loops=1
           Buffers: shared hit=5686
 Planning:
   Buffers: shared hit=4
 Planning Time: 0.526 ms
 Execution Time: 180.660 ms
(22 rows)

Index-Only Scan

An Index-Only Scan is the superstar of scans and answers the entire query using only the information stored within the index itself. Index only scans are also called “covering indexes” meaning the index itself covers all the data. It never even has to touch the main table. Index only scans are a huge performance win because they’re very fast - no information needs to be retrieved from the heap table. They also typically use less i/o resources because indexes are very cache friendly and often in shared buffers - meaning no data needs to be read for the underlying disk.

Queries benefit from a covering index in these situations:

• The query is very frequently executed.
• The current query is performing a standard index scan followed by many slow disk reads (heap fetches) and using i/o.
• The query only requires a small subset of the table's columns, for example you select only three columns from a table of twenty.
• The columns have a low write frequency. Any column that is indexed must be written to disk and the index, so if you start adding covering indexes for all your columns - you’re essentially creating write amplification.
• The new index, which must cover all needed columns, won't be excessively large. Indexes are stored on disk so you don’t want to cause storage issues.

EXPLAIN (ANALYZE, VERBOSE, BUFFERS)
SELECT code, status
FROM index_only_test
WHERE code > 'CODE_050000'
ORDER BY code
LIMIT 100;
                                                                           QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.42..2.60 rows=100 width=13) (actual time=0.346..0.362 rows=100.00 loops=1)
   Output: code, status
   Buffers: shared hit=1 read=3
   ->  Index Only Scan using idx_code_status on public.index_only_test  (cost=0.42..1068.02 rows=49000 width=13) (actual time=0.345..0.352 rows=100.00 loops=1)
         Output: code, status
         Index Cond: (index_only_test.code > 'CODE_050000'::text)
         Heap Fetches: 0
         Index Searches: 1
         Buffers: shared hit=1 read=3
 Planning:
   Buffers: shared hit=19
 Planning Time: 1.838 ms
 Execution Time: 0.385 ms
(13 rows)

Summary

We’ve covered all the major scan types so now reading your EXPLAIN plans will be a little easier.

• Seq scan - Postgres looks through the whole table in sequential order to find the query data
• Index scan - Postgres first looks at the index and then fetches the row data the index pointed to
• Bitmap index scan - Postgres first read the index and created a bitmap list matching rows. Second, Postgres read the data heap using the bitmap in a more efficient method than a sequential scan.
• Parallel scan - Postgres used multiple parallel workers to scan the table and data was gathered at the end
• Parallel index scan - Postgres used multiple workers to do an index scan and data was gathered at the end
• Index only scan- All data for the query was in the index

And here’s everything all in one graphic:

]]> 08ee92c4d2dfc4ad6be90a57493965c9cdb5a8e3c06cedc4fd8eddfb425c08c9 Thu, 04 Dec 2025 08:00:00 EST 2025-12-04T13:00:00.000Z 2025-12-04T13:00:00.000Z https://www.crunchydata.com/blog/postgres-internals-hiding-in-plain-sight Postgres has an awesome amount of data collected in its own internal tables. Postgres hackers know all about this  - but software developers and folks working with day to day Postgres tasks often miss out the good stuff.

The Postgres catalog is how Postgres keeps track of itself. Of course, Postgres would do this in a relational database with its own schema. Throughout the years several nice features have been added to the internal tables like psql tools and views that make navigating Postgres’ internal tables even easier.

Today I want to walk through some of the most important Postgres internal data catalog details. What they are, what is in them, and how they might help you understand more about what is happening inside your database.

psql’s catalog information

The easiest way to get at some of Postgres’ internal catalogs is to use the built-in psql commands that begin \d generally. Here’s some common Postgres ones users should be comfortable using:

\d {tablename}: describes a specific table. \d will do a lot of things if you qualify \d with a table or view name.

\di: list all your indexes

\dx: list installed extensions

\dp: to show access privileges

\dp+: tables and views with the roles and access details

\dconfig: your current configuration settings

\dt {tablename}: describe a table

\dti+: tables and indexes with sizes

\dg+: show role names

\df:  show your functions

\dv {view name}: describe a view

\l: lists all your databases

Important Postgres catalog views

Postgres exposes many of the complex internals of the database system in easy-to-query views. These host a wealth of information about what is going on inside your database and direct SQL access to answer in the moment emergency questions like “what is taking up all my CPU” and more long term questions like “what are my 10 slowest queries”.

pg_stat_activity

Shows current database activity, including running queries, state, and client information. Essential for troubleshooting and getting process ids (pid) for bad actors.

SELECT pid, usename, datname, client_addr, application_name, state, query
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY state, query_start DESC;

pg_stat_statements

This requires the pg_stat_statements extension - but it is part of the contrib library and ships with Postgres, so doesn't require separate installation.

This view tracks execution statistics for all queries executed by all databases. It's incredibly powerful for identifying slow or frequently executed queries.

-- pg_stat_statements 10 longest running queries
SELECT query, calls, total_exec_time, mean_exec_time, rows
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

pg_stat_database

This view provides database-wide statistics, such as the number of connections, transactions, and I/O. It's useful for a high-level overview of database activity and health.

-- high leve db stats for the postgres db
SELECT datname,numbackends, xact_commit, xact_rollback, blks_read, blks_hit
FROM pg_stat_database
WHERE datname = 'postgres';

pg_locks

This view displays information about locks held by active processes. This is the go to place for troubleshooting locking issues, deadlocks, and contention within the database. We have a great blog on locking and how to find the source of the lock in Postgres.

-- locks joined with the activity table. Shows not granted locks, typically those that could not be granted because they were blocked by other locks
SELECT a.datname, l.pid, l.locktype, l.relation::regclass, l.mode, l.granted
FROM pg_locks l
JOIN pg_stat_activity a ON l.pid = a.pid
WHERE NOT l.granted;

pg_stat_user_tables

This view offers statistics on tables, including sequential scans, index scans, and row-level operations (inserts, updates, deletes). It's great for identifying tables with heavy activity or those that need vacuuming.

-- see sequence scans and index scans by table
SELECT relname AS table_name, seq_scan, idx_scan
FROM pg_stat_user_tables
WHERE seq_scan > 0 OR idx_scan > 0 ORDER BY seq_scan DESC;

pg_stat_user_indexes

This view provides statistics on user indexes, such as how often they're used and how many tuples are read. This is particularly herpful for finding unused or underutilized indexes.

-- Never used indexes in Postgres sorted by size
SELECT s.schemaname, s.relname AS table_name, s.indexrelname AS index_name, pg_size_pretty(pg_relation_size(s.indexrelid)) AS index_size, s.idx_scan
FROM pg_stat_user_indexes AS s
JOIN pg_index AS i ON s.indexrelid = i.indexrelid
WHERE s.idx_scan = 0 AND i.indisunique IS FALSE
ORDER BY pg_relation_size(s.indexrelid) DESC;

pg_settings

This is a prebuilt view that is super useful for viewing configuration parameters, their current values, and their descriptions. Qualify with ILIKE to see exact parameters you’re looking for.

-- find shared_buffer or work_mem settings
SELECT name, setting, unit, short_desc
FROM pg_settings
WHERE name LIKE '%shared_buffers%' OR name LIKE '%work_mem%';

pg_roles

This view describes all system roles, which include users and groups. It's useful for checking permissions, login capabilities, and role memberships.

-- This query lists all roles, showing their names, whether they can log in, and their password expiration date.
SELECT rolname, rolcanlogin, rolvaliduntil
FROM pg_roles
ORDER BY rolname;

pg_database

This view contains all databases in the cluster. It provides key metadata for each database, including its owner, character encoding, and access privileges. We have a lot of folks now that create dozens and sometimes hundreds of databases for development, so this is a good high level view.

-- This query lists all Postgres databases, their sizes, and owners.
SELECT d.datname AS database_name, pg_size_pretty(pg_database_size(d.datname)) AS database_size, pg_get_userbyid(d.datdba) AS owner
FROM pg_database AS d
WHERE d.datistemplate = false;

Postgres catalog tables

Behind the Postgres metacommands and views - there are several core catalog tables. Many of the psql commands match up with the catalog tables. Something roughly like this:

psql command	what data	catalog tables
\d	tables and table objects	pg_class
\di	indexes	pg_class, pg_index
\dx	installed extensions	pg_extension
\dp	tables and privileges	pg_class, pg_roles, pg_attribute
\l	databases	pg_database
\df	available functions	pg_proc

Let’s look at these and how you might want to use them.

pg_stats

The pg_stats table collects all the details about your columns - things like cardinality - are there many items in this column or a few? Postgres uses a lot of the details in pg_stats to make decisions for the query planner and efficiently. In some cases, giving pg_stats more information can make your queries faster.

-- table column data like cardinality
SELECT * FROM pg_stats
WHERE tablename = 'table_name'
AND attname = 'column_name';

pg_class

pg_class contains a row for every table, index, sequence, view, materialized view, and other "relation-like" objects in the database. Sometimes this is a nice high level view of an entire table’s accoutrements.

SELECT c.relname, pg_get_userbyid(c.relowner) AS owner
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public' AND c.relkind = 'r'
ORDER BY c.relname;

pg_type

This table stores all data types that exist. It's confusing though - in Postgres, every table has an associated composite type that defines the structure of its rows. So if you do a select * you’ll see all the table names here and all the data types. If you filter a bit, you can see all your custom data types, domains, and enums.

-- see your custom data types in Postgres
SELECT
    t.typname AS type_name,
    n.nspname AS schema_name,
    t.typtype AS type_class
FROM
    pg_type AS t
JOIN
    pg_namespace AS n ON t.typnamespace = n.oid
LEFT JOIN pg_class c ON typrelid = c.oid
WHERE
    t.typtype IN ('e', 'd', 'c') -- 'e' for enum, 'd' for domain, 'c' for composite types.
    AND n.nspname NOT IN ('pg_catalog', 'information_schema', 'pg_toast')
    AND (t.typtype <> 'c' OR c.relkind = 'c')
ORDER BY
    schema_name, type_name;

pg_proc

This is the catalog of all functions and stored procedures that Postgres can use. It contains metadata about each routine. Made a function last week but can’t find it now? Just scan through all of them.

-- This query finds all functions, triggers, and stored procedures.
SELECT proname AS function_name, proargnames AS argument_names, pg_catalog.format_type(prorettype, NULL) AS return_type
FROM pg_proc
ORDER BY proname;

pg_attribute

This table stores information about table columns and there is one row in pg_attribute for every column in every table. While indexes and other objects that have an entry in pg_class.

Query columns and data types for any table with a query like this:

SELECT
    a.attname AS column_name,
    pg_catalog.format_type(a.atttypid, a.atttypmod) AS data_type
FROM
    pg_catalog.pg_attribute a
WHERE
    a.attrelid = 'orders'::regclass
    AND a.attnum > 0
    AND NOT a.attisdropped
ORDER BY
    a.attnum;

pg_catalog schema

The pg_catalog is the schema holding the system tables, so you will either need to include pg_catalog in your search_path (the default), or any query you issue will need to be qualified with pg_catalog.

Here’s a summary of the internal catalog tables:

pg_catalog	schema holding all the catalog tables
pg_stats	table and column statistics, like cardinality
pg_attribute	row for every table column
pg_class	every table, index, view, materialized view, forgien table
pg_type	data types, built in and custom

Exploring system tables with ECHO_HIDDEN or -E

Sometimes navigating these tables and views can be confusing and require browsing through a mix of docs and source code. If you want to have some fun exploring how the catalog is connected, you can connect to your database with -E argument to psql (or do \set ECHO_HIDDEN on if you’re already connected). Postgres will echo each psql the command that's run with SQL so you can grab the underlying SQL and edit from there.

For example, echoing \dt+ will show me a query and the results.

SELECT n.nspname as "Schema",
  c.relname as "Name",
  CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'm' THEN 'materialized view' WHEN 'i' THEN 'index' WHEN 'S' THEN 'sequence' WHEN 't' THEN 'TOAST table' WHEN 'f' THEN 'foreign table' WHEN 'p' THEN 'partitioned table' WHEN 'I' THEN 'partitioned index' END as "Type",
  pg_catalog.pg_get_userbyid(c.relowner) as "Owner",
  CASE c.relpersistence WHEN 'p' THEN 'permanent' WHEN 't' THEN 'temporary' WHEN 'u' THEN 'unlogged' END as "Persistence",
  am.amname as "Access method",
  pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as "Size",
  pg_catalog.obj_description(c.oid, 'pg_class') as "Description"
FROM pg_catalog.pg_class c
     LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
     LEFT JOIN pg_catalog.pg_am am ON am.oid = c.relam
WHERE c.relkind IN ('r','p','')
      AND n.nspname <> 'pg_catalog'
      AND n.nspname !~ '^pg_toast'
      AND n.nspname <> 'information_schema'
  AND pg_catalog.pg_table_is_visible(c.oid)
ORDER BY 1,2;

List of tables
-[ RECORD 1 ]-+--------------
Schema        | public
Name          | articles
Type          | table
Owner         | dba
Persistence   | permanent
Access method | heap
Size          | 16 kB
Description

Getting to Postgres internals

1. The easiest way to see internals is to start with the psql \d commands
2. The prebuilt views like pg_stat_activity, pg_stat_statements, pg_locks, and pg_stat_user_indexes are ready to go for easy querying and searching.
3. Going a step deeper, you can access the underlying internal Postgres tables, housed in the pg_catalog schema. -E echo_hidden can help you see the tables involved if you echo psql commands.

The PostgreSQL creators totally nailed it. They laid out a flexible framework for a variety of business use cases that would eventually become the most popular database 30 years later.

The paper outlines 6 project goals:

1. better support for complex objects growing world of business and engineering use cases

2. provide user extendibility for data types, operators and access methods

3. provide facilities for active databases like alerters and triggers

4. simplify process for crash recovery

5. take advantage of upgraded hardware

6. utilize Codd’s relational model

Let's look at all of them in reference to modern features of Postgres.

1) Objects and data types for a growing world of business and engineering use cases

Postgres has a rich and flexible set of native data types that are designed to meet a vast array of business use cases, from simple record-keeping to complex data analysis.

Numeric Types like SMALLINT and INTEGER are used for whole numbers while BIGINT might be for a user's unique ID or primary keys. Precision like NUMERIC and  DECIMAL are used, exact precision is critical, especially for money in Postgres. Floating-Point Types like REAL or DOUBLE PRECISION can be used for scientific or engineering calculations where absolute precision isn't as important as the range of values. You also have your UUID (indexable UUIDs in Postgres 18) for distributed systems and secure URLs.

Character Types like VARCHAR(n) or CHAR(n) store variable-length text up to a specified maximum length (n) and only use as much storage as needed for the actual text.

Date/Time Types like DATE stores only the date (year, month, day).  TIMESTAMPTZ is the time and date GOAT with and is easily implemented into global systems.

But, wait, that’s not all, Postgres has within it, the ability to easily make custom data types and constrain data to the specifics of each use case.

Using CREATE DOMAIN you can create specific value check like confirming a range for birthday or email format validity.

-- Postgres create domain
CREATE DOMAIN date_of_birth AS date
CHECK (value > '1930-01-01'::date);

CREATE DOMAIN valid_email AS text
NOT NULL
CHECK (value ~* '^[A-Za-z0-9._%-]+@[A-Za-z0-9.-]+[.][A-Za-z]+$');

Or using a direct CREATE TYPE you can make a new type as a composite. For example, new custom date type allowing for storage of height, width, and, weight in a single field.

-- Postgres create type with composite
CREATE TYPE physical_package AS (
height numeric,
width numeric,
weight numeric);

Enums let you create a custom type with a set of predefined values.

-- Postgres enum
CREATE TYPE order_status AS ENUM (
'pending',
'shipped',
'cancelled');

Constraints take the enumerated type a bit further and let you specify rules and restrictions for data. Additionally adding a CHECK constraint to a list or even refer to other fields, like reserving a room with a start and end time.

-- Postgres check contraint
ALTER TABLE public.reservations
ADD CONSTRAINT start_before_end
CHECK (start_time < end_time);

While most applications will constrain data in its own way, Postgres’ strict and flexible typing allows both rigid validity and flexibility.

2) Extensibility for data types, operators and access methods

The authors knew that just data types wouldn’t be enough - the system would actually need to be extensible. In my estimation - this is actually the killer feature of Postgres. Sure, the database is solid  - but the ingenuity and enthusiasm of the extension ecosystem is incredibly special.

Let’s take PostGIS for example. This extension adds several key data types to the mix - the point, line, polygon, to store geospatial types. PostGIS also has hundreds of functions with it. There’s now an entire ecosystem of its own around this project that includes open-source mapping and fully open source web servers that rival paid GIS systems like ESRI.

The pgvector extension is another good example of Postgres extensibility too. Now Postgres can store embedding data right alongside application data. You can have LLMs create embeddings based on your data and you can query your data to find relatedness. You can also build your own Postgres RAG system right inside your database

-- find distance between two embedding values
recipe_1.embedding <=> recipe_2.embedding

Data types and extensions aren’t the only thing that came out of this idea though - the indexes themselves in Postgres are incredibly advanced. Generalized Inverted Index (GIN) and Generalized Search Tree (GiST) are themselves extensible indexing frameworks that support many of the complex data types mentioned above.

3) Features for active databases like alerters and triggers

Modern Postgres users have a suite of tools available to them to have the database do necessary work. The trigger system easily updates fields once another field changes.

-- Postgres sample function to update fields
CREATE OR REPLACE FUNCTION update_inventory_on_sale()
RETURNS TRIGGER AS $$
BEGIN
UPDATE products
SET quantity_on_hand = quantity_on_hand - NEW.quantity_sold
WHERE id = NEW.product_id;
IF NOT FOUND THEN
RAISE EXCEPTION 'No product found with ID %', NEW.product_id;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;

For events outside the database, Postgres has a handy little NOTIFY/LISTEN mechanism for sending notifications to the outside so your application or dashboard will know when a new order was placed or a specific action happened. There’s an extension now to use the listen notify system events as WebSockets.

Postgres’ logical replication makes use of the ‘active database’ idea. PostgreSQL's logical replication is cool because it streams individual data changes rather than physical block-level copies, allowing you to replicate data between different major Postgres versions or even different platforms. This flexibility enables powerful use cases like creating specialized read replicas, consolidating multiple databases into a central one, and performing zero-downtime major version upgrades.

-- Postgres create logical replication
CREATE PUBLICATION user_pub FOR TABLE user_id, forum_posts;

4) Simplify process for crash recovery

The original method of Postgres data recovery relied on writing all data modifications to the files on disk before each commit which was called "force-to-disk". Unfortunately this original implementation had major performance issues and a potential for corruption. The Write Ahead Log (WAL) which was released with version 7.1 changed this into a different system that first writes changes to a log file and then applies those changes to the main data files.

WAL is the foundation of all of Postgres’ amazing backup and disaster recovery story. WAL is used to create incremental backups, complete with the Point-in-Time disaster recovery system that many rely on today.

WAL is also foundational to Postgres streaming replication, which makes high availability possible. A primary writes all database changes (inserts, updates, deletes) into its Write-Ahead Log and then "streams" these WAL records over the network to the standby (replica) nodes. The standby nodes receive these WAL records and apply them to their own copy of the database, keeping them in sync with the primary. In the event of an emergency automated failover, like Patroni, can promote a new primary.

5) Take advantage of upgraded hardware

PostgreSQL was engineered for the hardware realities of its time: single-core CPUs, severely limited RAM often measured in megabytes, and slow, spinning hard drives. The primary design focus was on correctness and data durability over raw speed. PostgreSQL built its legendary reputation for stability and ACID compliance, ensuring that data remained safe even when running on less reliable hardware.

Fast forward to today, where PostgreSQL runs on hardware with dozens of CPU cores, terabytes of ultra-fast NVMe storage and vast amounts of RAM (we even have half a tb of RAM available now). PostgreSQL recently introduced parallel query execution which breaks up complex queries and runs them simultaneously, gathering the results at the end. Modern PostgreSQL has also vastly improved its locking mechanisms, connection pooling solutions, and replication capabilities, evolving from a robust single-server database into a high-performance powerhouse that can scale horizontally and handle the massive, concurrent workloads of the modern internet.

While Postgres today does not yet have the modern CPU multi-threading, this is on the horizon, and Postgres 18 just added asynchronous i/o.

6) Utilize Codd’s relational model

At the height of the NoSQL movement in the late 2000s and early 2010s, a common story was told that relational databases were a relic of the past. With the rise of big and unstructured data, this old model may soon be cast out.

Postgres continued to do what it always has done and embraced its core strength - flexibility of data typing – and adopted some of NoSQL’s own ideas. Postgres introduced the JSON data type and then later the binary, indexable JSONB type. With this update, applications can now store schema-less API driven JSON data directly in a relational database and query it efficiently using a rich set of operators and functions. With features like json_table, you can go between arrays or traditional tables.

The newest revolution in the Postgres world seems to be the adoption of technologies to tie Postgres directly to unstructured flat files. Projects like pg_duckdb, pg_mooncake, and Crunchy Data Warehouse use custom extensions to work directly with files in csv, Parquet, and Iceberg directly in the data lake remote object stores where they reside. Even with the data abstracted to another location, Postgres’ relational model is still relevant, efficient, and trusted.

Summary

With Postgres’ flexibility - you can have a fully normalized, relational schema with foreign keys and JOINs, while also having an indexed JSONB document and full spatial geometry. We’re at a point in history where AI, science, and research are backed by a database that had no idea what the world would be like when it was built. Postgres is still here.

These original goals have had a profound impact on the project. Allowing for complexity and flexibility in a growing business landscape, while being easy to alter for individual use cases. And being ready for hardware (and cloud) technology that makes Postgres’ distribution even easier. ]]> 1e8a25ee9384198a8eb616a19c72c4d852e451563c2209f890ca1e7d8545a4ac Tue, 23 Sep 2025 09:00:00 EDT 2025-09-23T13:00:00.000Z 2025-09-23T13:00:00.000Z https://www.crunchydata.com/blog/get-excited-about-postgres-18 Postgres 18 will be released in just a couple weeks! Here’s some details on the most important and exciting features.

Asynchronous i/o

Postgres 18 is adding asynchronous i/o. This means faster reads for many use cases. This is also part of a bigger series of performance improvements planned for future Postgres, part of which may be multi-threading. Expect to see more on this in coming versions.

What is async I/O?

When data isn’t in the shared memory buffers already, Postgres reads from disk, and I/O is needed to retrieve data. Synchronous I/O means that each individual request to the disk is waited on for completion before moving on to something else. For busy databases with a lot of activity, this can be a bottleneck.

Postgres 18 will introduce asynchronous I/O, allowing workers to optimize idle time and improve system throughput by batching reads. Currently, Postgres relies on the operating system for intelligent I/O handling, expecting OS or storage read-ahead for sequential scans and using features like Linux's posix_fadvise for other read types like Bitmap Index Scans. Moving this work into the database with asynchronous I/O will provide a more predictable and better-performing method for batching operations at the database level. Additionally, a new system view, pg_aios, will be available to provide data about the asynchronous I/O system.

Postgres writes will continue to be synchronous - since this is needed for ACID compliance.

If async i/o seems confusing, think of it like ordering food at a restaurant. In a synchronous model, you would place your order and stand at the counter, waiting, until your food is ready before you can do anything else. In an asynchronous model, you place your order, receive a buzzer, and are free to go back to your table and chat with friends until the buzzer goes off, signaling that your food is ready to be picked up.

Async I/O will affect:

• sequential scans
• bitmap heap scans (following the bitmap index scan)
• some maintenance operations like VACUUM.

By default Postgres will turn on io_method = worker. By default there are 3 workers and this can be adjusted up for systems with larger CPU workers. I haven’t seen any reliable recommendations on this, so stay tuned for more on that from our team soon.

For Postgres running on Linux 5.1+ you can utilize the io_uring system calls and have the invocations made via the actual backends rather than having separate processes with the optional io_method = io_uring.

UUID v7

UUIDs are getting a bit of an overhaul in this version by moving to v7.

UUIDs are randomly generated strings which are globally unique and often used for primary keys. UUIDs are popular in modern applications for a couple reasons:

• They’re unique: You can use keys generated from more than one place.
• Decoupled:Your application can generate a primary key before sending the data to the database.
• URL obscurity: If your URLs use primary keys (e.g., .../users/5), other URLs are easy to guess (.../users/6, .../users/7). With a UUID (.../users/f47ac10b-58cc-4372-a567-0e02b2c3d479), it's impossible to guess other IDs.

A new standard for UUID v7 came out in mid-2024 via a series of standards updates. UUIDv4 was the prior version of uuid with native Postgres support. But sorting and indexing in large tables had performance issues due to the relative randomness, leading to fragmented indexes and bad locality.  UUIDv7 helps with the sort and indexing issues. It is still random but that first 48 bits (12 characters) are a timestamp, and the remaining bits are random; this gives better locality for data inserted around the same time and thus better indexability.

The timestamp part is a hexadecimal value (i.e. compressed decimal). So for example a uuid that begins with 01896d6e4a5d6 (hex) would represent the 2707238289622 (decimal) and that is the number of milliseconds since 1970.

This is how the DDL will look for uuid v7:

CREATE TABLE user_actions (
action_id UUID PRIMARY KEY DEFAULT uuidv7(),
user_id BIGINT NOT NULL,
action_description TEXT,
action_time TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_action_id ON user_actions (action_id);

B-tree skip scans

There’s a nice performance bump coming in Postgres 18 for some multi-column B-tree indexes.

In Postgres, if you have an index on columns (status, date) in a table, this index can be used to match queries which query both status and date fields, or just status.

In Postgres 17 and below, this same index cannot be used to answer queries against just the date field; you would have to have that column indexed separately or the database would resort to a sequence scan + filter approach if there were no appropriate indexes for that table.

In Postgres 18, in many cases it can automatically use this multi-column index for queries touching only the date field.  Known as a skip scan, this lets the system "skip" over portions of the index.

This works when queries don’t use the leading columns in the conditions and the omitted column has a low cardinality, like a small number of distinct values. The optimization works by:

1. Identifying all the distinct values in the omitted leading column(s).
2. Effectively transform the query to add the conditions to match the leading values.
3. The resulting query is able to use existing infrastructure to optimize lookups across multiple leading columns, effectively skipping any pages in the index scan which do not match both conditions.

For example, if we had a sales table with columns status and date, we might have a multi-column index:

CREATE INDEX idx_status_date
ON sales (status, date);

An example query could have a where clause that doesn’t include status.

SELECT * FROM sales
WHERE date = '2025-01-01';

Nothing in the query plan tells you this is a skip scan, so you’ll end up with a normal Index scan like this, showing you the index conditions.

                                QUERY PLAN
-------------------------------------------------------------
 Index Only Scan using idx_status_date on sales  (cost=0.29..21.54 rows=4 width=8)
   Index Cond: (date = '2025-01-01'::date)
(2 rows)

Before 18, a full table scan would be done, since the leading column of the index is not included, but with skip scan Postgres can use the same index for this index scan.

In Postgres 18, because status has a low cardinality and just a few values, a compound index scan can be done. Note that this optimization only works for queries which use the = operator, so it will not work with inequalities or ranges.

This all happens behind-the-scenes in the Postgres planner so you don’t need to turn it on. The idea is that it will benefit analytics use cases where filters and conditions often change and aren’t necessarily related to existing indexes.

The query planner will decide if using a skip scan is worthwhile, based on the table's statistics and the number of distinct values in the columns being skipped.

Generated columns on-the-fly

PostgreSQL 18 introduces virtual generated columns. Previously, generated columns were always stored on disk. This meant for generated columns, values were computed at the time of an insert or update and adding a bit of write overhead.

In PostgreSQL 18, virtual generated columns are now the default type for generated columns. if you define a generated column without explicitly specifying STORED, it will be created as a virtual generated column.

CREATE TABLE user_profiles (
user_id SERIAL PRIMARY KEY,
settings JSONB,
username VARCHAR(100) GENERATED ALWAYS AS (settings ->> 'username') VIRTUAL
);

This is a great update for folks using JSON data, queries can be simplified and data changes or normalization can be done on the fly as needed.

Note that virtual generated columns are not indexable - since they’re not stored on disk. For indexing of JSONB, use the stored version or expression index.

OAUTH 2.0

Good news for folks that use Okta, Keycloak, and other managed authentication services, Postgres is now compatible with OAUTH 2.0. This is specified in the main host based authentication configuration (pg_hba.conf) file.

The Oauth system uses bearer tokens where the client application presents a token instead of a password to prove identity. The token is an opaque string and its format is determined by the authorization server. This feature removes the need to store passwords in the database. It also allows for more robust security measures like multi-factor authentication (MFA) and single sign-on (SSO) to be managed by external identity providers.

Postgres versions are packed with other improvements

Postgres 18 comes with a staggering 3,000 commits from more than 200 authors. While many of these are features, there are numerous additions and optimizations under the hood to the Postgres query planner and other parts of the system that are behind the scenes. Even if you don’t utilize optional features, there’s still performance benefits (uh ... asyc i/o is a biggie), bug fixes, and security patches that make upgrading on a regular cadence a good idea. ]]> 0fe99b43c2417b308d641253451cc38618f70b171a295266a2dd8108b823f133 Fri, 12 Sep 2025 08:00:00 EDT 2025-09-12T12:00:00.000Z 2025-09-12T12:00:00.000Z