Huge Pages and Postgres in Containers
We recently participated in a community solution for using huge pages when you’re running Postgres in containers or with Crunchy Postgres for Kubernetes. We worked on a patch to the underlying OCI (Open Container Initiative) runtime specification with our partner Red Hat and also worked on a patch for Postgres 16. For those of you using huge pages or running in containers, we have some additional notes on our solution in this write up. We’re really proud of the improvements we’ve made because they help Postgres, Kubernetes, and every container runtime!
Background on Huge Pages
CPUs translate virtual memory addresses to physical addresses in chunks called “pages.” Pages are typically 4 KB each, but nearly all CPU architectures provide a way to use larger sizes, often 2 MB or 1 GB. Those larger pages are called “huge pages” in Linux and are more efficient when using lots of memory. Huge pages can improve Postgres performance and protect Postgres background processes from the Out Of Memory (OOM) manager. Anyone who adjusts Postgres shared_buffers
should consider tuning their system’s huge pages.
Because huge pages are so great, Crunchy Postgres for Kubernetes makes them super easy to use in the resources
portion of the PostgresCluster YAML. The following example starts Postgres with 10 gigs of memory, 2 of which are huge pages. Kubernetes finds a machine the right size, and Postgres uses what’s available.
apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
name: pg
spec:
postgresVersion: 14
backups:
pgbackrest:
repos:
- name: repo1
volume:
volumeClaimSpec:
accessModes: [ReadWriteOnce]
resources: { requests: { storage: 1Gi } }
instances:
- dataVolumeClaimSpec:
accessModes: [ReadWriteOnce]
resources: { requests: { storage: 1Gi } }
resources:
requests:
cpu: 2
memory: 8Gi
limits:
hugepages-2Mi: 2Gi
Huge pages missing in container runtimes
Crunchy Postgres for Kubernetes initially released this feature in 2021. Every once in a while, we would get a report that Postgres could not initialize due to Bus error
, indicating that huge pages are to blame. The reporter would change their environment or set huge_pages = off
and be satisfied. Earlier this year, we decided to dig in and really identify what was going on.
We reproduced the issue and saw that the container’s hugetlb.2MB.limit_in_bytes
cgroup matched the Kubernetes hugepages-2Mi
limit, but hugetlb.2MB.rsvd.limit_in_bytes
did not. That difference explained the error, and setting the latter field made everything work again. (That made sense because the rsvd
field was added to the kernel because no one liked the bus errors caused by the other field.)
Surely, we thought, something in the tall stack between Kubernetes and Linux was supposed to configure this field. Which one was misbehaving and deserved a bug report? None of them, it turns out! Container runtimes didn’t set this field because v1.0 of the OCI Runtime Specification didn’t mention it at all.
Naturally, we were not the first to notice this. We found issues related to this error and these cgroup fields scattered across Postgres, Kubernetes, operators, containers, and container runtimes for years. None, however, were making any progress toward a solution.
Odin Ugedal reported the problem to OCI and suggested a solution in 2020. Kailun Qin submitted a patch in 2021. We reviewed this patch and enlisted the help of our partner, Red Hat, to get it merged. It is now part of OCI Runtime Specification v1.1 released in July. Look for it to be in container runtimes soon! 🎉
Postgres 16 allows huge pages during initialization
While we were working on OCI, David Angel, a PGO user, engaged with the pgsql-bugs mailing list after struggling to avoid the issue on a system with huge pages. As a result, our own Tom Lane added a feature to Postgres 16 allowing server variables to be set using initdb. With that, initdb with --set huge_pages=off
works on any system where huge pages are broken for any reason.
Better huge pages in the future for everyone
These two changes above are proper long-term solutions, but they’ll take time to make their way into your environments. In the meantime, Crunchy Postgres for Kubernetes has implemented workarounds that will keep you running smoothly with huge pages.
Related Articles
- Postgres Partitioning with a Default Partition
16 min read
- Iceberg ahead! Analyzing Shipping Data in Postgres
8 min read
- PostGIS Day 2024 Summary
8 min read
- Crunchy Data Warehouse: Postgres with Iceberg for High Performance Analytics
8 min read
- Loading the World! OpenStreetMap Import In Under 4 Hours
6 min read