Not too long ago I wrote a blog post about how to deploy TLS for Postgres on Kubernetes in attempt to provide a helpful guide from bringing your own TLS/PKI setup to Postgres clusters on Kubernetes. In part, I also wanted a personal reference for how to do it!
However, some things have changed since I first wrote that post. OpenSSL released a fix for CVE-2021-3450 (courtesy to my colleague Tom Swartz for reminding me of this) that prevents users from bypassing some of the x509 certificate verifications. This rolled out in various updates to OpenSSL and has found its way into PGO, the open source Postgres Operator for Kubernetes from Crunchy Data.
For PGO functionality, the OpenSSL fix has zero effect: PGO is already following proper x509 enforcement guidance. However, you can see the effects of this when using Postgres itself: Postgres uses OpenSSL to handle TLS and certificate-based authentication. An improper certificate will now fail verifications.
How to Setup TLS for Postgres in Kubernetes: CVE-2021-3450 Edition
Before reading this, I suggest making yourself familiar with the previous blog post in this unintended series. Like the previous blog post, I am going to generate a ECDSA certificate.
Unlike the previous blog post, to demonstrate the difference, I am also going to set up certificate-based authentication between Postgres instances using PGO, the Postgres Operator. Using the previous recipe with newer versions of OpenSSL, you would not be able to set this up.
The first step to get this working is to copy your template
file. This file is used to provide information about the x509 v3 extensions,
which will be required later on in the tutorial. As of this writing on EL8 (RHEL
8 / CentOS 8), you can find this file at
/etc/pki/tls/openssl.cnf. I copied it
into my working directory:
cp /etc/pki/tls/openssl.cnf ./openssl.cnf
Open up this file. You are going to want to check two things:
x509_extensionsis uncommented and set to
req_extensionsis uncommented and set to
Also check that the
[ v3_req ] block contains at least these values:
[ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment
Generating the Certificate Authority (CA)
From here, let's create a CA. Borrowing from the previous recipe, you can run this command to create your CA, with a few slight modifications:
openssl req \ -x509 \ -nodes \ -newkey ec \ -pkeyopt ec_paramgen_curve:prime256v1 \ -pkeyopt ec_param_enc:named_curve \ -sha384 \ -keyout ca.key \ -out ca.crt \ -days 3650 \ -extensions v3_ca \ -subj "/CN=\*"
Based on your OpenSSL configuration the
-extensions v3_ca may be superfluous.
If you inspect the CA certificate with:
openssl x509 -noout -text -in ca.crt
You should see something similar to:
Certificate: Data: Version: 3 (0x2) Serial Number: 04:c0:65:62:89:55:ff:3f:38:db:88:37:1f:2e:2e:aa:8e:68:ce:92 Signature Algorithm: ecdsa-with-SHA384 Issuer: CN = * Validity Not Before: May 11 15:10:16 2021 GMT Not After : May 9 15:10:16 2031 GMT Subject: CN = * Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:0b:cc:86:af:44:de:cd:d4:f4:9f:96:01:d2:11: 5e:c2:be:46:6a:79:3b:88:cd:a0:94:38:ac:0a:f9: 13:51:e9:1c:70:a1:68:a5:2d:03:8d:01:b8:27:d8: 83:95:ec:ef:c1:ea:74:e5:b1:65:55:84:18:9e:64: 7e:f2:19:44:48 ASN1 OID: prime256v1 NIST CURVE: P-256 X509v3 extensions: X509v3 Subject Key Identifier: 6D:A9:B1:FD:13:18:13:DD:7D:58:ED:47:AA:26:FD:43:4C:3B:5C:B3 X509v3 Authority Key Identifier: keyid:6D:A9:B1:FD:13:18:13:DD:7D:58:ED:47:AA:26:FD:43:4C:3B:5C:B3 X509v3 Basic Constraints: critical CA:TRUE Signature Algorithm: ecdsa-with-SHA384 30:45:02:20:08:93:06:bf:b0:94:a1:8d:8e:09:3a:4d:75:13: 8c:b7:d0:b3:a1:f4:54:50:9e:94:c7:20:c9:f9:51:77:1c:c7: 02:21:00:b1:c9:30:d4:90:dd:ea:38:0b:9c:bd:05:1b:e9:71: 75:49:ec:c1:af:d4:83:23:d9:a2:2d:3e:83:ac:27:fb:8a
The main thing to note is:
X509v3 Basic Constraints: critical CA:TRUE
which indicates that this certificate represents a certificate authority (CA).
Generating the Server Certificate
Now, let's generate the server certificate. This is where the recipe diverges from the last time.
First, generate the certificate signing request (CSR) that we will use for our CA to sign the certificate:
openssl req \ -new \ -newkey ec \ -nodes \ -pkeyopt ec_paramgen_curve:prime256v1 \ -pkeyopt ec_param_enc:named_curve \ -sha384 \ -keyout server.key \ -out server.csr \ -days 365 \ -subj "/CN=hippo.pgo"
Now, let's have the CA create the certificate. This time, we will explicitly
pass in the
v3_req extension to ensure that the key uses are present:
openssl x509 \ -req \ -in server.csr \ -days 365 \ -CA ca.crt \ -CAkey ca.key \ -CAcreateserial \ -sha384 \ -extfile openssl.cnf \ -extensions v3_req \ -out server.crt
Let's inspect the server certificate:
openssl x509 -noout -text -in server.crt
Should yield something similar to:
Certificate: Data: Version: 3 (0x2) Serial Number: 3f:c7:d8:57:76:3e:05:c6:ea:dc:e2:b9:58:08:3f:a3:de:8d:02:97 Signature Algorithm: ecdsa-with-SHA384 Issuer: CN = * Validity Not Before: May 11 15:15:35 2021 GMT Not After : May 11 15:15:35 2022 GMT Subject: CN = hippo.pgo Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:7a:f5:54:b5:0f:b2:ae:c7:b4:55:f6:65:97:19: f2:eb:ab:bf:c5:2b:ef:ee:12:4e:82:3b:4b:5f:90: 96:89:9c:4a:64:3f:9b:e6:c1:19:bc:18:80:cf:ab: e6:0e:a9:7c:35:bf:4a:8e:e6:3d:78:32:ec:1a:8e: 32:25:ac:75:8b ASN1 OID: prime256v1 NIST CURVE: P-256 X509v3 extensions: X509v3 Basic Constraints: CA:FALSE X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment Signature Algorithm: ecdsa-with-SHA384 30:46:02:21:00:ab:0c:3d:cb:e2:38:e9:66:86:3c:ee:5f:ca: c7:d4:3d:db:e0:f4:e7:cd:64:e8:ff:dc:cf:53:58:8b:3c:fe: 81:02:21:00:b6:2f:a2:20:f5:5a:73:c5:49:9d:8c:5b:e4:52: 28:bb:9c:d3:13:e9:57:6f:d9:05:45:68:37:65:54:13:22:0d
Again, here is the main part to notice:
X509v3 Basic Constraints: CA:FALSE X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment
The main part to notice is the x509 v3 key usages section. The previous recipe did not set this; a main component of CVE 2021-3450 is that OpenSSL is ensuring that the key usages are checked when strict verification is enabled (which is a good idea). By using the above method, we can ensure that we are generating certificates that are prepared for newer versions of OpenSSL.
To really see this in action in the context of PGO and Postgres, let's generate a certificate for replica authentication.
Generating the Replica Authentication Certificate
The formula for generating the replica authentication certificate is similar for the server certificate. The two noticeable differences:
- The name of the file is prefixed with
CNwill have the name
primaryuser; this is to conveniently map to what PGO configures in Postgres for certificate-based authentication between replicas.
Here is the recipe below:
openssl req \ -new \ -newkey ec \ -nodes \ -pkeyopt ec_paramgen_curve:prime256v1 \ -pkeyopt ec_param_enc:named_curve \ -sha384 \ -keyout replicas.key \ -out replicas.csr \ -days 365 \ -subj "/CN=primaryuser" openssl x509 \ -req \ -in replicas.csr \ -days 365 \ -CA ca.crt \ -CAkey ca.key \ -CAcreateserial \ -sha384 \ -extfile openssl.cnf \ -extensions v3_req \ -out replicas.crt
Feel free to inspect the certificate with
openssl x509 -noout -text -in replicas.crt to ensure that the appropriate x509
v3 key usages are set.
Let's now tie this into creating a Postgres cluster.
Creating a TLS Enforced Postgres Cluster with Certificate Authentication Between Replicas
(Say that section heading ten times fast!)
This recipe will assume that you have already
to Kubernetes and you are working in a namespace named
First, we need to create a few Secrets for the CA, the server certificate, and the replica certificate. We can do that with the following commands:
kubectl create secret generic -n pgo postgres-ca --from-file=ca.crt=ca.crt kubectl create secret tls -n pgo hippo.tls --key=server.key --cert=server.crt kubectl create secret tls -n pgo hippo-replicas.tls --key=replicas.key --cert=replicas.crt
Once those are created, let's create a cluster that enforces TLS and includes TLS authentication between replicas:
pgo create cluster hippo \ --tls-only \ --replica-count=1 \ --server-ca-secret=postgres-ca \ --server-tls-secret=hippo.tls \ --replication-tls-secret=hippo-replicas.tls
You may need to wait a few moments for everything to provision.
How will we know that everything worked? For one, your replica instance will be connected, healthy, and receiving changes from the primary. We can verify this in a few ways.
First, inspect the logs on the replica:
kubectl -n jkatz logs --selector=pg-cluster=hippo,role=replica --tail=100
You should see something similar to this (some output truncated)
# ... pg_hba: - hostssl replication primaryuser 0.0.0.0/0 cert # ... 2021-05-11 15:27:37,204 INFO: replica has been created using pgbackrest 2021-05-11 15:27:37,209 INFO: bootstrapped from leader 'hippo-778c9bd7b5-hnz44' # ... 2021-05-11 15:27:40,488 INFO: establishing a new patroni connection to the postgres cluster 2021-05-11 15:27:40,565 INFO: no action. i am a secondary and i am following a leader # ...
This logs indicate that we are able to connect to a primary, authenticated via certificates.
Open source and all of its dependencies can present fun challenges to ensuring your environment still works when software updates occur, particularly when the update deals with closing a CVE. It's important to continuously test your environment to ensure it is still working as expected...just as its important for recipe writers to continuously test their recipes!
Jonathan S. Katz
May 13, 2021 •More by this author