Postgres for analytics has always been a huge question mark. By using PostgreSQL's extension APIs, integrating DuckDB as a query engine for state-of-the-art analytics performance without forking either project could Postgres be the analytics database too?

Bringing an analytical query engine into a transactional database system raises many interesting possibilities and questions. In this blog post I want to reflect on what makes these workloads and system architectures so different and what bringing them together means.

OLAP & OLTP: Never the twain shall meet

There’s a couple super common Postgres errors you’re likely to encounter while using this database, especially with an application or ORM. One is the PG::DiskFull: ERROR: could not resize shared memory segment. It will look something like this.

"PG::DiskFull: ERROR: could not resize shared memory segment "/PostgreSQL.938232807" to 55334241 bytes: No space left on device"

Back in the 1990s, before anything was cool (or so my children tell me) and at the dawn of the Age of the Meme, a couple of college students invented a game they called the "Six Degrees of Kevin Bacon".

The conceit behind the Six Degrees of Kevin Bacon

One of the major changes that the cloud brought to application and database management was the concept of "thin provisioning." With large amounts of compute or storage resources available behind an API, you can provision what you need now and expand your infrastructure as required. Frameworks like 12Factor

If you missed the database news lately, you could have missed that we just fused DuckDB with Postgres to build a really fast analytics platform based on Postgres.

There’s so many interesting things you can do with this platform so expect to hear from me again 😉. Today I just want to show off one really simple trick for getting big data sets or training data into Postgres through Hugging Face.

Hugging Face

As I’ve been working with Postgres psql cli, I’ve picked up a few good habits from my Crunchy Data co-workers that make my terminal database environment easier to work with. I wanted to share a couple of my favorite things I’ve found that make getting around Postgres better. If you’re just getting started with psql, or haven’t ventured too far out of the defaults, this is the post for you. I’ll walk you through some of the friendliest psql settings and how to create your own preset settings file.

Some of the most helpful psql commands

We recently introduced support for querying Iceberg tables from PostgreSQL in Crunchy Bridge for Analytics. Iceberg defines a way to store tables in data lakes (usually as Parquet files in S3) with support for snapshots and other important database features, and it is designed with high performance analytics in mind.

If you’re new to Crunchy Bridge, it offers a fully managed PostgreSQL

In April we launched Crunchy Bridge for Analytics, which is a managed PostgreSQL option that enables fast and seamless querying of your data lake. Our initial release was focused on building a rock solid foundation for high performance analytics in PostgreSQL. We have since been hard at work turning it into a comprehensive analytics solution.

Our goals in building Crunchy Bridge for Analytics are to:

• Make it very easy to query data files (incl. Parquet/CSV/JSON/Iceberg) in object stores like S3 from PostgreSQL, as well as easy data import/export.
• Offer best-in-class analytics performance, for example by integrating DuckDB into PostgreSQL

Many folks are surprised to hear that Postgres has parallel queries out of the box. This was released in small batches across a half dozen versions of Postgres, so the major fanfare for having parallelism got a little bit lost.

By default Postgres is configured for two parallel workers. The Postgres query planner will assemble a few plans for any given query and will estimate the additional overhead of performing parallel queries, and make a go or no-go decision. Depending on the settings and the calculations of the query planner, parallel queries are typically used by large and long running queries — like warehouse or analytical workloads.

Below is the output of a sample EXPLAIN

Crunchy Data is pleased to announce the publication of the Crunchy Data PostgreSQL 16 Security Technical Implementation Guide