<?xml version="1.0" encoding="UTF-8" ?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" version="2.0"><channel><title>Bob Pacheco | CrunchyData Blog</title>
<atom:link href="https://www.crunchydata.com/blog/author/bob-pacheco/rss.xml" rel="self" type="application/rss+xml" />
<link>https://www.crunchydata.com/blog/author/bob-pacheco</link>
<image><url>https://www.crunchydata.com/build/_assets/bob-pacheco.png-7HBAVBAP.webp</url>
<title>Bob Pacheco | CrunchyData Blog</title>
<link>https://www.crunchydata.com/blog/author/bob-pacheco</link>
<width>1004</width>
<height>888</height></image>
<description>PostgreSQL experts from Crunchy Data share advice, performance tips, and guides on successfully running PostgreSQL and Kubernetes solutions</description>
<language>en-us</language>
<pubDate>Wed, 17 Jan 2024 08:00:00 EST</pubDate>
<dc:date>2024-01-17T13:00:00.000Z</dc:date>
<dc:language>en-us</dc:language>
<sy:updatePeriod>hourly</sy:updatePeriod>
<sy:updateFrequency>1</sy:updateFrequency>
<item><title><![CDATA[ Troubleshooting Postgres in Kubernetes ]]></title>
<link>https://www.crunchydata.com/blog/troubleshooting-postgres-in-kubernetes</link>
<description><![CDATA[ Bob helps you fix some common Kubernetes installation issues. He shows you the right order of things, how to fix a missing image, and some common resource allocation errors. ]]></description>
<content:encoded><![CDATA[ <p>In my role as a Solutions Architect at Crunchy Data, I help customers get up and running with <a href=https://www.crunchydata.com/products/crunchy-postgresql-for-kubernetes>Crunchy Postgres for Kubernetes</a> (CPK). Installing and managing a Postgres cluster in Kubernetes has never been easier. However, sometimes things don't go as planned and I’ve noticed a few major areas where Kubernetes installations go awry. Today I want to walk through some of the most common issues I see when people try to get up and running with Postgres in Kubernetes and offer a list of basic troubleshooting ideas to get started. Now sure, your issue might not be in here, but if you’re just trying to diagnose a bad install or a failing cluster, here’s my go to list of where to get started<h2 id=the-order-of-things-crd-operator-cluster-pod><a href=#the-order-of-things-crd-operator-cluster-pod>The Order of Things: CRD, Operator, Cluster, Pod</a></h2><p>Let’s get started with a basic understanding of how things get installed and by what. You can use that knowledge to determine where to look first when something that you are expecting does not appear during your installation.<p><strong>Custom Resource Definition (CRD)</strong>: The CPK Operator requires a Custom Resource Definition (CRD). It is possible to have multiple CRDs per Operator. Our most recent Operator, 5.5, has 3 CRD examples, postgres-operator.crunchydata.com_postgresclusters.yaml being one of them. The user applies all CRD files to the Kubernetes cluster. The CRDs must be installed before the operator.<p><strong>Operator</strong>: The CPK Operator gets installed by the user applying a manager.yaml file that describes a Kubernetes object of <a href=https://kubernetes.io/docs/concepts/workloads/controllers/deployment/>kind:Deployment</a>. This creates the Deployment and the Deployment creates the Operator pod. The Operator itself is a container running in a pod.<p><strong>Postgres Cluster</strong>: A CPK Postgres Cluster is typically created by the user applying a postgres.yaml file containing the PostgresCluster.spec, that describes a Kubernetes object of <a href=https://access.crunchydata.com/documentation/postgres-operator/latest/references/crd/5.5.x/postgrescluster>kind:PostgresCluster</a>.<p><strong>Pods</strong>: The stateful sets and deployments create the individual pods they describe. The Operator creates a stateful set for each Postgres pod and the pgBackRest repo host pod (if applicable). Deployments are also created for pgBouncer pods (if applicable). If you are missing a pod, describe the Deployment or StatefulSet that owns it. If you are missing a Deployment or StatefulSet, the CPK Operator logs will usually indicate why.<h2 id=image-pulls><a href=#image-pulls>Image Pulls</a></h2><p>Next, let’s look at image pull issues. There are two primary reasons why you would receive an image pull error. 1 - you do not have permissions to connect to the registry or pull the requested image. Or 2 - the image requested is not in the registry.<h3 id=permissions-example><a href=#permissions-example>Permissions Example</a></h3><p>I am attempting to deploy the CPK Operator.<pre><code class=language-bash>kubectl apply -n postgres-operator -k install/default --server-side

</code></pre><p>I see that I have an ImagePullBackOff error.<pre><code class=language-bash>kubectl -n postgres-operator get pods
NAME                   READY   STATUS             RESTARTS   AGE
pgo-5694b9545c-ggz7g   0/1     ImagePullBackOff   0          27s

</code></pre><p>When looking at issues with a pod not coming up in Kubernetes the first thing we will do is run a describe on the pod and look at the events in the bottom of the output.<pre><code class=language-bash>kubectl -n postgres-operator describe pod pgo-5694b9545c-ggz7g
...
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  ...
  Normal   Pulling    6m9s (x4 over 7m39s)    kubelet            Pulling image "registry.crunchydata.com/crunchydata/postgres-operator:ubi8-5.5.0-0"
  Warning  Failed     6m9s (x4 over 7m39s)    kubelet            Failed to pull image "registry.crunchydata.com/crunchydata/postgres-operator:ubi8-5.5.0-0": rpc error: code = Unknown desc = failed to pull and unpack image "registry.crunchydata.com/crunchydata/postgres-operator:ubi8-5.5.0-0": failed to resolve reference "registry.crunchydata.com/crunchydata/postgres-operator:ubi8-5.5.0-0": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to &#60https://access.crunchydata.com/api/v1/auth/jwt/container/token/?scope=repository%3Acrunchydata%2Fpostgres-operator%3Apull&#38service=crunchy-container-registry:> 403 Forbidden

</code></pre><p>Looking in the events, we see that we attempted to pull the crunchydata/postgres-operator:ubi8-5.5.0-0 pod from the Crunchy Data registry. We see in the next event entry: 403 Forbidden. This means that we do not have the permissions to pull this pod from this registry.<p><strong>Adding a pull secret</strong><p>To resolve the issue, we will create a pull secret and add it to the deployment. You can find more information on creating pull secrets for private registries in the <a href=https://access.crunchydata.com/documentation/postgres-operator/latest/installation/private-registries>CPK documentation</a>.<p>We created the image pull secret and added it to the deployment per the documentation. We apply the change and delete the failed pod. Now we see that the pod is recreated and the image is pulled successfully.<pre><code class=language-bash>kubectl apply -n postgres-operator -k install/default --server-side

kubectl -n postgres-operator delete pod pgo-5694b9545c-xnpjg
pod "pgo-5694b9545c-xnpjg" deleted

kubectl -n postgres-operator get pods
NAME                   READY   STATUS    RESTARTS   AGE
pgo-5694b9545c-xnpjg   1/1     Running   0          23s

</code></pre><h3 id=image-not-in-registry-example><a href=#image-not-in-registry-example>Image Not In Registry Example</a></h3><p>We again attempt to deploy the Operator and see that we have an ImagePullBackOff error.<pre><code class=language-bash>kubectl -n postgres-operator get pods
NAME                   READY   STATUS             RESTARTS   AGE
pgo-6bfc9554b7-6h4jd   0/1     ImagePullBackOff   0          22s

</code></pre><p>Just like before, we will describe the pod and look at the events to determine why this is happening:<pre><code class=language-bash>kubectl -n postgres-operator describe pod pgo-6bfc9554b7-6h4jd

...
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
...
  Normal   Pulling    4m30s (x4 over 6m5s)  kubelet            Pulling image "registry.crunchydata.com/crunchydata/postgres-operator:ubi8-5.50.0-0"
  Warning  Failed     4m30s (x4 over 6m4s)  kubelet            Failed to pull image "registry.crunchydata.com/crunchydata/postgres-operator:ubi8-5.50.0-0": rpc error: code = NotFound desc = failed to pull and unpack image "registry.crunchydata.com/crunchydata/postgres-operator:ubi8-5.50.0-0": failed to resolve reference "registry.crunchydata.com/crunchydata/postgres-operator:ubi8-5.50.0-0": registry.crunchydata.com/crunchydata/postgres-operator:ubi8-5.50.0-0: not found

</code></pre><p>This time we see that we tried to pull the crunchydata/postgres-operator:ubi8-5.50.0-0 image from the Crunchy Data registry. However, the image is not found. Upon closer inspection of the image listed in the CPK Operator kustomization.yaml file we see that we have a typo. We had a tag of ubi8-5.50.0-0 when it should have been ubi8-5.5.0-0.<pre><code class=language-yaml>images:
  - name: postgres-operator
    newName: registry.crunchydata.com/crunchydata/postgres-operator
    newTag: ubi8-5.50.0-0
</code></pre><p><strong>Changing tag names</strong><p>We make the correction to the file and apply the change. The pod is automatically recreated with the correct image tag.<pre><code class=language-bash>kubectl apply -n postgres-operator -k install/default --server-side

kubectl -n postgres-operator get pods
NAME                   READY   STATUS    RESTARTS   AGE
pgo-6bfc9554b7-6h4jd   1/1     Running   0          96s

</code></pre><p>By using the Kubernetes describe pod function we were able to see why we were getting image pull errors and easily correct them.<h2 id=resource-allocation><a href=#resource-allocation>Resource Allocation</a></h2><p>Another important place to look when troubleshooting a failed Kubernetes installation is looking at resource allocations, and ensuring pods have the necessary CPU and memory. The most common issues I see at installation time are:<ul><li>Requesting more resources then are available in the available Kubernetes nodes.<li>Insufficient resource requests to allow for the proper operation of the containers running in the pod.</ul><h3 id=resource-request-exceeds-availability><a href=#resource-request-exceeds-availability>Resource Request Exceeds Availability</a></h3><p>Here in this postgres.yaml we set some resource requests and limits for our Postgres pods. We are requesting 5 CPUs and setting a limit of 10 CPUs per Postgres pod.<pre><code class=language-yaml>instances:
  - name: pgha1
    replicas: 2
    resources:
      limits:
        cpu: 10000m
        memory: 256Mi
      requests:
        cpu: 5000m
        memory: 100Mi
</code></pre><p>When we create the Postgres cluster and look at the pods we find them in a pending state.<pre><code class=language-bash>kubectl apply -n postgres-operator -k high-availability
postgrescluster.postgres-operator.crunchydata.com/hippo-ha created

</code></pre><pre><code class=language-bash>kubectl -n postgres-operator get pods                                                                                                       ──(Tue,Dec19)─┘
NAME                                 READY   STATUS    RESTARTS   AGE
hippo-ha-pgbouncer-7c467748d-tl4pn   2/2     Running   0          103s
hippo-ha-pgbouncer-7c467748d-v6s4d   2/2     Running   0          103s
hippo-ha-pgha1-bzrb-0                0/5     Pending   0          103s
hippo-ha-pgha1-z7nl-0                0/5     Pending   0          103s
hippo-ha-repo-host-0                 2/2     Running   0          103s
pgo-6ccdb8b5b-m2zsc                  1/1     Running   0          48m

</code></pre><p>Let's describe one of the pending pods and look at the events:<pre><code class=language-bash>kubectl -n postgres-operator describe pod hippo-ha-pgha1-bzrb-0
Name:             hippo-ha-pgha1-bzrb-0
Namespace:        postgres-operator
...
Events:
  Type     Reason             Age                    From                Message
  ----     ------             ----                   ----                -------
  Warning  FailedScheduling   3m41s (x2 over 3m43s)  default-scheduler   0/2 nodes are available: 2 Insufficient cpu. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod..

</code></pre><p>We see that there is insufficient available CPU to meet our request. We reduce our resource request and limits and try again.<pre><code class=language-yaml>instances:
  - name: pgha1
    replicas: 2
    resources:
      limits:
        cpu: 1000m
        memory: 256Mi
      requests:
        cpu: 500m
        memory: 100Mi
</code></pre><pre><code class=language-bash>kubectl apply -n postgres-operator -k high-availability
postgrescluster.postgres-operator.crunchydata.com/hippo-ha created

</code></pre><pre><code class=language-bash>kubectl -n postgres-operator get pods                                                                                                       ──(Tue,Dec19)─┘
NAME                                 READY   STATUS    RESTARTS   AGE
hippo-ha-backup-jb8t-tgdtx           1/1     Running   0          13s
hippo-ha-pgbouncer-7c467748d-s8wq6   2/2     Running   0          34s
hippo-ha-pgbouncer-7c467748d-zhcmf   2/2     Running   0          34s
hippo-ha-pgha1-hmrq-0                5/5     Running   0          35s
hippo-ha-pgha1-xxtf-0                5/5     Running   0          35s
hippo-ha-repo-host-0                 2/2     Running   0          35s
pgo-6ccdb8b5b-m2zsc                  1/1     Running   0          124m

</code></pre><p>Now we see that all of our pods are running as expected.<h3 id=insufficient-resource-request><a href=#insufficient-resource-request>Insufficient Resource Request</a></h3><p>What happens if we don't allocate enough resources? Here we set very low CPU requests and limits. We are requesting 5m CPUs and setting a limit of 10m CPUs.<pre><code class=language-yaml>instances:
  - name: pgha1
    replicas: 2
    resources:
      limits:
        cpu: 10m
        memory: 256Mi
      requests:
        cpu: 5m
        memory: 100Mi
</code></pre><p>We apply the manifest and take a look at the pods.<pre><code class=language-bash>kubectl apply -n postgres-operator -k high-availability
postgrescluster.postgres-operator.crunchydata.com/hippo-ha created

</code></pre><pre><code class=language-bash>kubectl -n postgres-operator get pods
NAME                                 READY   STATUS    RESTARTS      AGE
hippo-ha-pgbouncer-7c467748d-hnf5k   2/2     Running   0             93s
hippo-ha-pgbouncer-7c467748d-q28t9   2/2     Running   0             93s
hippo-ha-pgha1-r2qs-0                4/5     Running   2 (11s ago)   93s
hippo-ha-pgha1-x2ft-0                4/5     Running   2 (8s ago)    93s
hippo-ha-repo-host-0                 2/2     Running   0             93s
pgo-6ccdb8b5b-m2zsc                  1/1     Running   0             136m

</code></pre><p>We see that our Postgres pods are only showing 4/5 containers running and 90 seconds after creation they have already restarted twice. This is a clear indication that something is wrong. Let's look at the logs for the Postgres container to see what is going on.<pre><code class=language-bash>kubectl -n postgres-operator logs hippo-ha-pgha1-r2qs-0 -c database

</code></pre><p>We didn't get any logs back. This indicates that the Postgres container is not starting. Now we will adjust the CPU request and limit to more reasonable values and try again. I normally don’t go below 500m.<pre><code class=language-yaml>instances:
  - name: pgha1
    replicas: 2
    resources:
      limits:
        cpu: 1000m
        memory: 256Mi
      requests:
        cpu: 500m
        memory: 100Mi
</code></pre><pre><code class=language-bash>kubectl apply -n postgres-operator -k high-availability
postgrescluster.postgres-operator.crunchydata.com/hippo-ha created

</code></pre><p>Now we see that our cluster is up and running with all expected containers.<pre><code class=language-bash>kubectl -n postgres-operator get pods
NAME                                 READY   STATUS    RESTARTS   AGE
hippo-ha-backup-pv9n-tr7mh           1/1     Running   0          6s
hippo-ha-pgbouncer-7c467748d-45jj9   2/2     Running   0          33s
hippo-ha-pgbouncer-7c467748d-lqfz2   2/2     Running   0          33s
hippo-ha-pgha1-8kh2-0                5/5     Running   0          34s
hippo-ha-pgha1-v4t5-0                5/5     Running   0          34s
hippo-ha-repo-host-0                 2/2     Running   0          33s
pgo-6ccdb8b5b-m2zsc                  1/1     Running   0          147m

</code></pre><h2 id=storage-allocation><a href=#storage-allocation>Storage Allocation</a></h2><p>Lastly, we will look at some common issues when allocating storage to our pods. The most common issues that someone will run into regarding storage allocation at installation time are:<ul><li>Improper Resource Request<li>Unsupported Storage Class</ul><h3 id=improper-resource-request-example><a href=#improper-resource-request-example>Improper Resource Request Example</a></h3><p>Here is an example of the storage we want to allocate to our Postgres cluster pods in the postgres.yaml:<pre><code class=language-yaml>dataVolumeClaimSpec:
  accessModes:
    - 'ReadWriteOnce'
  resources:
    requests:
      storage: 1GB
</code></pre><p>When we attempt to apply the manifest we see this output on the command line:<pre><code class=language-bash>k apply -n postgres-operator -k high-availability
The PostgresCluster "hippo-ha" is invalid: spec.instances[0].dataVolumeClaimSpec.resources.requests.storage: Invalid value: "1GB": spec.instances[0].dataVolumeClaimSpec.resources.requests.storage in body should match '^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$'

</code></pre><p>The value of "1GB" is invalid. The error message tells you where in the manifest the error is. It is in the <code>spec.instances[0].dataVolumeClaimSpec.resources.requests.storage</code> section of the manifest. The message even provides the regex that is used for validation.<p>When we enter the valid value of <code>1Gi</code> we are able to deploy our Postgres cluster. Remember that gigabytes must be described in Gi, and megabytes with Mi. More syntax specifics in the <a href=https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes>Kubernetes docs</a>.<pre><code class=language-yaml>dataVolumeClaimSpec:
  accessModes:
    - 'ReadWriteOnce'
  resources:
    requests:
      storage: 1Gi
affinity:
</code></pre><pre><code class=language-bash>kubectl -n postgres-operator get pods
NAME                                 READY   STATUS    RESTARTS   AGE
hippo-ha-backup-ngg5-56z7z           1/1     Running   0          10s
hippo-ha-pgbouncer-7c467748d-4q887   2/2     Running   0          35s
hippo-ha-pgbouncer-7c467748d-lc2sr   2/2     Running   0          35s
hippo-ha-pgha1-w9vc-0                5/5     Running   0          35s
hippo-ha-pgha1-zhx8-0                5/5     Running   0          35s
hippo-ha-repo-host-0                 2/2     Running   0          35s
pgo-6ccdb8b5b-vzzkp                  1/1     Running   0          12m

</code></pre><h3 id=improper-storage-class-name-example><a href=#improper-storage-class-name-example>Improper Storage Class Name Example</a></h3><p>We want to specify a specific storage class to be used with our Postgres cluster pods:<pre><code class=language-yaml>dataVolumeClaimSpec:
  storageClassName: foo
  accessModes:
    - 'ReadWriteOnce'
  resources:
    requests:
      storage: 1Gi
</code></pre><p>When we apply the manifest we see that our Postgres pods get stuck in a "pending" state.<pre><code class=language-bash>kubectl -n postgres-operator get pods
NAME                                 READY   STATUS    RESTARTS   AGE
hippo-ha-pgbouncer-7c467748d-jxxpf   2/2     Running   0          3m42s
hippo-ha-pgbouncer-7c467748d-wdtvq   2/2     Running   0          3m42s
hippo-ha-pgha1-79gr-0                0/5     Pending   0          3m42s
hippo-ha-pgha1-xv2t-0                0/5     Pending   0          3m42s
hippo-ha-repo-host-0                 2/2     Running   0          3m42s
pgo-6ccdb8b5b-vzzkp                  1/1     Running   0          24m

</code></pre><p>At this point it is not clear to us why the pods are pending. Let's describe one of them and look at the events to see if we can get more information.<pre><code class=language-bash>kubectl -n postgres-operator describe pod hippo-ha-pgha1-79gr-0
Name:             hippo-ha-pgha1-79gr-0
Namespace:        postgres-operator
...
Events:
  Type     Reason             Age                   From                Message
  ----     ------             ----                  ----                -------
  Normal   NotTriggerScaleUp  31s (x32 over 5m34s)  cluster-autoscaler  pod didn't trigger scale-up:
  Warning  FailedScheduling   13s (x6 over 5m36s)   default-scheduler   0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..

</code></pre><p>In the describe events we see that the pod has unbound immediate PersistentVolumeClaims. What does that mean? It means that Kubernetes was not able to meet our storage claim request so it remains unbound. If we examine our dataVolumeClaimSpec we see that we set three specific values:<pre><code class=language-yaml>dataVolumeClaimSpec:
  storageClassName: foo
  accessModes:
    - 'ReadWriteOnce'
  resources:
    requests:
      storage: 1Gi
</code></pre><p>We review the available storage classes in our Kubernetes provider. In this case we are deploying on GKE. We see that we have 3 storage classes available to us:<p><img alt=storage_classes.png loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/79f28a80-978a-4281-3894-ed71a9c5ac00/public><p>We delete the failed cluster deployment<pre><code class=language-bash>kubectl delete -n postgres-operator -k high-availability
postgrescluster.postgres-operator.crunchydata.com "hippo-ha" deleted

</code></pre><p>We update the storageClassName in our manifest to a supported storage class and apply it.<pre><code class=language-yaml>dataVolumeClaimSpec:
  storageClassName: standard-rwo
  accessModes:
    - 'ReadWriteOnce'
  resources:
    requests:
      storage: 1Gi
</code></pre><pre><code class=language-bash>kubectl apply -n postgres-operator -k high-availability
configmap/db-init-sql created
postgrescluster.postgres-operator.crunchydata.com/hippo-ha created

</code></pre><p>Now we see that all of our pods are up and running.<pre><code class=language-bash>kubectl -n postgres-operator get pods
NAME                                 READY   STATUS    RESTARTS   AGE
hippo-ha-backup-jstq-c8n67           1/1     Running   0          6s
hippo-ha-pgbouncer-7c467748d-5smt9   2/2     Running   0          31s
hippo-ha-pgbouncer-7c467748d-6vb7t   2/2     Running   0          31s
hippo-ha-pgha1-9s2g-0                5/5     Running   0          32s
hippo-ha-pgha1-drmv-0                5/5     Running   0          32s
hippo-ha-repo-host-0                 2/2     Running   0          32s
pgo-6ccdb8b5b-vzzkp                  1/1     Running   0          44m

</code></pre><h2 id=we-did-it><a href=#we-did-it>We Did It!</a></h2><p>In this blog we were able to identify, diagnose and correct common installation issues that sometimes occur when installing Postgres in Kubernetes. We learned how to use the Kubernetes describe function to obtain information that assisted us in the diagnosis of the issues we ran into. The lessons learned here don't just apply to Postgres. These types of issues can happen with any application running in Kubernetes if the manifest is not correct or proper resources have not been allocated. Congratulations! You now have the knowledge you need to solve common installation issues. ]]></content:encoded>
<category><![CDATA[ Kubernetes ]]></category>
<author><![CDATA[ Bob.Pacheco@crunchydata.com (Bob Pacheco) ]]></author>
<dc:creator><![CDATA[ Bob Pacheco ]]></dc:creator>
<guid isPermalink="false">2926e3265fa1a428e81d271c8507ddaf9c4decf2696d6f8548db39e669d6fba7</guid>
<pubDate>Wed, 17 Jan 2024 08:00:00 EST</pubDate>
<dc:date>2024-01-17T13:00:00.000Z</dc:date>
<atom:updated>2024-01-17T13:00:00.000Z</atom:updated></item>
<item><title><![CDATA[ Deploying Crunchy Postgres for Kubernetes in a Multi-Zone Cluster ]]></title>
<link>https://www.crunchydata.com/blog/deploying-crunchy-postgres-for-kubernetes-in-a-multi-zone-cluster</link>
<description><![CDATA[ Bob has all the instructions for getting your Postgres in a Kubernetes multi-AZ environment. ]]></description>
<content:encoded><![CDATA[ <p>Modern resilient data infrastructure ensures that there's not a single point of failure. In the cloud, this means eliminating single points of failure at the data center and availability zone. So when deploying <a href=https://www.crunchydata.com/blog/database-terminology-explained-postgres-high-availability-and-disaster-recovery>highly available</a> Postgres in Kubernetes, you will want to ensure that each availability zone has one or more Postgres pods running in it. Today I want to walk through the basics of setting up a multi-zone Kubernetes implementation using <a href=https://www.crunchydata.com/products/crunchy-postgresql-for-kubernetes>Crunchy Postgres for Kubernetes</a>. We will use a combination of PostgresCluster.spec.instances and <a href=https://www.crunchydata.com/blog/kubernetes-pod-tolerations-and-postgresql-deployment-strategies>node affinity</a> to enable this functionality.<p>To avoid performance issues, it is generally a good idea to not create Kubernetes clusters across regions. The regions have not been optimized for high-bandwidth, low-latency network connections for cross region communications. Instead, you should consider creating your kubernetes clusters across multiple availability zones within the same region and use a <a href=https://access.crunchydata.com/documentation/postgres-operator/latest/architecture/disaster-recovery>stand-by CPK cluster</a> in a different region to solve the geographical single point of failure problem.<h3 id=quick-terminology-refresher><a href=#quick-terminology-refresher>Quick terminology refresher</a></h3><ul><li>Multi-Zone Cluster - A kubernetes cluster that spans multiple availability zones, sometimes called a stretched cluster.<li>Regions - Regions are collections of zones. Zones have high-bandwidth, low-latency network connections to other zones in the same region.<li>Availability Zones - A zone is a deployment area within a region.</ul><h2 id=the-kubernetes-cluster><a href=#the-kubernetes-cluster>The Kubernetes cluster</a></h2><p>Using GKE, I setup a cluster with 1 kubernetes node in 3 availability zones: <img alt=multi-zone-nodes loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/608d7c6f-3f2a-45b8-b977-6de94b2e8300/public><p><strong>Note:</strong> As a prerequisite, I installed the latest version of the CPK Operator in my kubernetes cluster. Steps to do that can be found <a href=https://access.crunchydata.com/documentation/postgres-operator/latest/quickstart>at the quickstart</a>.<h2 id=working-with-postgrescluster-instances><a href=#working-with-postgrescluster-instances>Working with PostgresCluster instances</a></h2><p>The Postgres cluster spec supports a collection of instances. An instance is a grouping of Postgres pods where you can define:<ul><li>replica counts<li>resources<li>affinities<li>taints</ul><p>A typical postgres cluster will have only one instance defined with multiple replicas. However, for this scenario, we will create an instance for each availability zone within our kubernetes cluster.<h2 id=the-custom-resource><a href=#the-custom-resource>The Custom Resource</a></h2><p>We create one instance in our PostgresCluster.spec.instances collection for each availability zone that we want to deploy our postgres pods into. We set the instance name to align with the availability zone name so we can determine that our pod assignments are working as expected. We set 1 replica per instance and add a node affinity section to the instance to associate each instance with the desired availability zone. Our PostgresCluster custom resource looks like this:</p><details><summary>- ha-multi-zone.yaml</summary><pre><code class=language-yaml>apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: hippo-ha
spec:
  image: registry.crunchydata.com/crunchydata/crunchy-postgres:ubi8-15.4-0
  imagePullSecrets:
    - name: crunchyreg
  postgresVersion: 15
  instances:
    - name: zone-a
      replicas: 1
      dataVolumeClaimSpec:
        accessModes:
          - 'ReadWriteOnce'
        resources:
          requests:
            storage: 1Gi
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels:
                    postgres-operator.crunchydata.com/cluster: hippo-ha
                    postgres-operator.crunchydata.com/instance-set: zone-a
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: topology.kubernetes.io/zone
                    operator: In
                    values:
                      - us-central1-a
    - name: zone-b
      replicas: 1
      dataVolumeClaimSpec:
        accessModes:
          - 'ReadWriteOnce'
        resources:
          requests:
            storage: 1Gi
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels:
                    postgres-operator.crunchydata.com/cluster: hippo-ha
                    postgres-operator.crunchydata.com/instance-set: zone-b
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: topology.kubernetes.io/zone
                    operator: In
                    values:
                      - us-central1-b
    - name: zone-c
      replicas: 1
      dataVolumeClaimSpec:
        accessModes:
          - 'ReadWriteOnce'
        resources:
          requests:
            storage: 1Gi
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels:
                    postgres-operator.crunchydata.com/cluster: hippo-ha
                    postgres-operator.crunchydata.com/instance-set: zone-c
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: topology.kubernetes.io/zone
                    operator: In
                    values:
                      - us-central1-c
  backups:
    pgbackrest:
      image: registry.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-2.47-0
      repos:
        - name: repo1
          volume:
            volumeClaimSpec:
              accessModes:
                - 'ReadWriteOnce'
              resources:
                requests:
                  storage: 1Gi
</code></pre></details>
</br><p>We have 3 instances:<ul><li>zone-a<li>zone-b<li>zone-c</ul><p>We deploy our postgres cluster to our multi-zone cluster using our custom resource:<pre><code class=language-bash>kubectl apply -n postgres-operator -k ha-multi-zone
</code></pre><p>Once the cluster is up and running we can check our postgres pod host assignments:<pre><code class=language-bash>robertpacheco@Roberts-MBP kustomize % kubectl -n postgres-operator get po -o wide
NAME                                  READY   STATUS    RESTARTS   AGE    IP           NODE                                                  NOMINATED NODE   READINESS GATES
hippo-ha-repo-host-0                  2/2     Running   0          2m8s   10.36.0.5    gke-pachecob-multi-zone-c-default-pool-194c2c58-5jpf   &#60none>           &#60none>
hippo-ha-zone-a-ft66-0                4/4     Running   0          2m8s   10.36.0.4    gke-pachecob-multi-zone-c-default-pool-194c2c58-5jpf   &#60none>           &#60none>
hippo-ha-zone-b-cmtc-0                4/4     Running   0          2m8s   10.36.2.4    gke-pachecob-multi-zone-c-default-pool-271e87ab-w82c   &#60none>           &#60none>
hippo-ha-zone-c-qwf6-0                4/4     Running   0          2m8s   10.36.1.17   gke-pachecob-multi-zone-c-default-pool-a7a8b474-m117   &#60none>           &#60none>
pgo-6f49bf65c4-thw2m                  1/1     Running   0          3d1h   10.36.1.10   gke-pachecob-multi-zone-c-default-pool-a7a8b474-m117   &#60none>           &#60none>
</code></pre><p>We can compare them to our node list: <img alt=node-list loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/621b5c73-8133-4fb3-64b5-99d6e6cb0b00/public><p>We see that the zone-a pod is running on the zone a cluster node. The zone-b and zone-c pods are also running on the correct nodes.<p>Now we can verify that our pod will not be moved into another availability zone if it loses its host. I will go into the Google Cloud console and cordon one of my kubernetes nodes. This will make that node unschedulable for any pods. In this case I chose to cordon the zone-b node.<p>I then delete the zone-b postgres pod. When the reconciler recreates it the pod will go into and stay in a pending state until its host node becomes available again.<pre><code class=language-bash>kubectl delete -n postgres-operator pod hippo-ha-zone-b-cmtc-0
pod "hippo-ha-zone-b-cmtc-0" deleted
</code></pre><p>Notice that hippo-ha-zone-b-cmtc-0 is now in a pending state.<pre><code class=language-bash>kubectl -n postgres-operator get pods
NAME                                  READY   STATUS      RESTARTS   AGE
hippo-ha-repo-host-0                  2/2     Running     0          14m
hippo-ha-zone-a-ft66-0                4/4     Running     0          14m
hippo-ha-zone-b-cmtc-0                0/4     Pending     0          30s
hippo-ha-zone-c-qwf6-0                4/4     Running     0          14m
pgo-6f49bf65c4-thw2m                  1/1     Running     0          3d1h
</code></pre><p>If we describe that pod and look at the events in the output we see why:<pre><code class=language-bash>kubectl describe -n postgres-operator pod hippo-ha-zone-b-cmtc-0
...
Events:
  Type     Reason             Age    From                Message
  ----     ------             ----   ----                -------
  Warning  FailedScheduling   2m19s  default-scheduler   0/3 nodes are available: 1 node(s) were unschedulable, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..
  Warning  FailedScheduling   2m17s  default-scheduler   0/3 nodes are available: 1 node(s) were unschedulable, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..
  Normal   NotTriggerScaleUp  2m17s  cluster-autoscaler  pod didn't trigger scale-up:
</code></pre><p>We see that 0/3 nodes are available: 1 node(s) were unschedulable, 2 node(s) didn't match Pod's node affinity/selector. This is exactly the behavior we are expecting. Our node affinity is working as expected.<p>Now we can uncordon the zone-b node. Once I do, I see that the hippo-ha-zone-b-cmtc-0 pod is now scheduled on the node in zone-b again and it is up and running.<pre><code class=language-bash>kubectl -n postgres-operator get pods
NAME                                  READY   STATUS      RESTARTS   AGE
hippo-ha-repo-host-0                  2/2     Running     0          20m
hippo-ha-zone-a-ft66-0                4/4     Running     0          20m
hippo-ha-zone-b-cmtc-0                4/4     Running     0          6m22s
hippo-ha-zone-c-qwf6-0                4/4     Running     0          20m
pgo-6f49bf65c4-thw2m                  1/1     Running     0          3d1h
</code></pre><h2 id=summary><a href=#summary>Summary</a></h2><p><a href=https://www.crunchydata.com/products/crunchy-postgresql-for-kubernetes>Crunchy Postgres for Kubernetes</a> is incredibly versatile. Today we learned how to take advantage of that versatility to deploy highly available postgres clusters on a multi-zone kubernetes cluster. This technique eliminates the possibility of a single point of failure at the availability zone level. Check out our other technical deep dives about Kubernetes <a href=https://www.crunchydata.com/blog/topic/kubernetes>on our blog</a>. ]]></content:encoded>
<category><![CDATA[ Kubernetes ]]></category>
<author><![CDATA[ Bob.Pacheco@crunchydata.com (Bob Pacheco) ]]></author>
<dc:creator><![CDATA[ Bob Pacheco ]]></dc:creator>
<guid isPermalink="false">52a16652db90de3768f91d8c5201157d454fb4e33c525ceda79a1a9d3e47af9b</guid>
<pubDate>Tue, 24 Oct 2023 09:00:00 EDT</pubDate>
<dc:date>2023-10-24T13:00:00.000Z</dc:date>
<atom:updated>2023-10-24T13:00:00.000Z</atom:updated></item>
<item><title><![CDATA[ Use CI/CD to Automatically Update Postgres Images with Argo CD ]]></title>
<link>https://www.crunchydata.com/blog/use-ci-cd-to-automatically-update-postgres-images-with-argo-cd</link>
<description><![CDATA[ Bob continues his series on using Argo CD to create GitOps workflows with Crunchy PostgreSQL for Kubernetes. In this post, he shows you how to create automated image updates. You can get a new Postgres image all the way from source to tested and ready for production with this workflow automation. ]]></description>
<content:encoded><![CDATA[ <p>When working with containers you always have to be mindful of the age of the containers. Every day new CVEs are being discovered and are turning up in image scans. One benefit of having a CI/CD pipeline is the ability to implement security automation. Let's assume you release a monthly update of your containers that are built on the latest version of the base image and all of the most recent patches have been applied. This ensures that each month you can remediate any CVEs that might have popped up in your images since their initial release. In this blog we show you how to use ARGO CD Image Updater as part of your CI/CD pipeline to automatically deploy, test, and promote your updated images. All by doing nothing more than putting them into your registry.<p><img alt=workflow loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/ae3aa734-8a62-4939-6272-a141fe0a0e00/public><p>This is part 2 of CI/CD with Crunchy Postgres for Kubernetes and Argo series. We will pick up from where we left off in <a href=https://www.crunchydata.com/blog/ci-cd-with-crunchy-postgres-for-kubernetes-and-argo>part 1</a>. We will use Argo CD Image Updater to monitor a private Docker registry for changes to the Postgres image tag. The image updater will update the image tag in GitHub and the Argo CD application will deploy those changes to the postgres-dev namespace. Once deployed, the <a href=https://github.com/bobpach/Crunchy-Postgres-Self-Test>self-test</a> will run and the changes will be applied to the postgres-qa namespace if all tests pass.<h3 id=prerequisites><a href=#prerequisites>Prerequisites</a></h3><p>There are a few prerequisites you will need to handle if you plan on following along with this example :<ul><li>A fully functional Argo CD deployment and a Crunchy Data Postgres cluster as described in my previous <a href=https://www.crunchydata.com/blog/ci-cd-with-crunchy-postgres-for-kubernetes-and-argo>CI/CD blog</a>.<li>A private container registry containing the images you want to deploy. Most organizations will pull images, tag them and then upload them into their private registries. For this blog I am using a private registry for all images except the self test. That image is in a public repo in my Docker registry.<li>An <a href=https://docs.docker.com/docker-hub/access-tokens/>access token</a> for your private registry.<li>A git repository containing the Crunchy Postgres for Kubernetes manifest to be deployed. Here's a sample manifest you can use or you can fork <a href=https://github.com/crunchydata/postgres-ci-cd-demo/>my git repository</a>.<li>A <a href=https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys>deploy key</a> with write access to your git repo.</ul><h3 id=secrets><a href=#secrets>Secrets</a></h3><p>We will need to create some secrets for the registry access token and git deploy key in the argocd namespace. Sample files are provided in my <a href=https://github.com/CrunchyData/postgres-ci-cd-demo/tree/main/Part-2-Update>git repository</a>. You will need to provide relevant values in the provided sample files and apply them to the argocd namespace.<pre><code class=language-shell>kubectl apply -n argocd -f secrets/privaterepo.yaml
kubectl apply -n argocd -f secrets/privatereg.yaml
</code></pre><p>You should already have a secret called argocd-token in the postgres-dev and postgres-qa namespaces. This secret contains the base64 encoded JWT token that was created in the sync role in the cicd project in ArgoCD. It was created as part of the initial CI/CD blog. If you do not have the secret, you can create it now:<pre><code class=language-shell>kubectl apply -n argocd -f secrets/argocd_token.yaml
</code></pre><h2 id=argo-cd-image-updater><a href=#argo-cd-image-updater>Argo CD Image Updater</a></h2><p><a href=https://argocd-image-updater.readthedocs.io/en/stable/>Argo CD Image Updater</a> is a tool to automatically update the container images of Kubernetes workloads that are managed by Argo CD. We will use it to monitor Postgres container images in our private Docker registry and update our image tag in our git repo.<h3 id=installation><a href=#installation>Installation</a></h3><p>We will install Argo CD Image Updater into the argocd namespace in our Kubernetes cluster. We already have Argo CD installed there from the previous blog.<pre><code class=language-shell>kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml
</code></pre><p>You should now see it in your pod list:<pre><code class=language-shell>$ kubectl -n argocd get po
NAME                                               READY  STATUS  RESTARTS  AGE
argocd-application-controller-0                    1/1    Running   0       80m
argocd-applicationset-controller-685679ccb9-x9962  1/1    Running   0       80m
argocd-dex-server-8f9dbcfb6-zrhgp                  1/1    Running   0       80m
argocd-image-updater-56d94c674d-8ddpf              1/1    Running   0       38s
argocd-notifications-controller-5d65949b4b-spp8p   1/1    Running   0       80m
argocd-redis-77bf5b886-67s9t                       1/1    Running   0       80m
argocd-repo-server-5b889d7495-r9clr                1/1    Running   0       80m
argocd-server-785bc6f697-pb6cn                     1/1    Running   0       80m
</code></pre><p>We need to inform the argocd-image-updater container about the location of the private registry that it will be monitoring for updates. We can do this by adding the following to the data property in the argocd-image-updater-config configmap in the argocd namespace:<pre><code class=language-yaml>data:
  registries.conf: |
    registries:
    - name: Docker Hub
     prefix: docker.io
     api_url: https://registry-1.docker.io
     credentials: pullsecret:argocd/privatereg
     defaultns: library
     default: true
</code></pre><p>We also need to assign policy information for the image updater. We can do this by adding the following to the data property in the argocd-rbac-cm configmap in the argocd namespace:<pre><code class=language-yaml>data:
  policy.csv: |
    p, role:image-updater, applications, get, */*, allow
    p, role:image-updater, applications, update, */*, allow
    g, image-updater, role:image-updater
  policy.default: role:readonly
</code></pre><h2 id=argo-cd-applications><a href=#argo-cd-applications>Argo CD Applications</a></h2><p>In the previous CI/CD blog we created two applications:<ul><li>postgres-dev<li>postgres-qa</ul><h3 id=annotations><a href=#annotations>Annotations</a></h3><p>We want Argo CD Image Updater to update our kustomization file in git with new image tags for images that it finds in our private repo. In order to do that we will have to add annotations in the postgres-dev application.<p>In the ARGO CD UI click on applications in the left pane. Click on the postgres-dev application. Click on APP Details in the top bar. Click edit in the top right of the application pane. Click the + button to add a new annotation. Enter the key and value. Do this for each annotation as listed below.<ul><li>argocd-image-updater.argoproj.io/image-list: postgres=&#60your_registry_name>/&#60your_image_name><li>e.g: argocd-image-updater.argoproj.io/image-list: postgres=bobpachcrunchy/crunchy-postgres<li>argocd-image-updater.argoproj.io/postgres.update-strategy: latest<li>argocd-image-updater.argoproj.io/write-back-method: git<li>argocd-image-updater.argoproj.io/git-branch: main<li>argocd-image-updater.argoproj.io/write-back-target: kustomization</ul><p><img alt=Annotations loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/180d984d-4f9f-40d0-7eb6-32418ec36f00/public><p>Click Save.<p>Note: In this demo we are only monitoring one image. You can monitor multiple images by adding them to the image-list annotation. See <a href=https://argocd-image-updater.readthedocs.io/en/stable/configuration/images/>Argo CD Image Updater doc</a> for more information.<h2 id=kustomize><a href=#kustomize>Kustomize</a></h2><p>In order for Argo CD Image Updater to update our images in git we need to change how we reference our image tags. In the previous CI/CD blog we referenced them in the PostgresCluster custom resource itself. Now we will move the Postgres image tag into the kustomization.yaml file. We will add a transformer and remove the tag from the custom resource.<h3 id=transformer><a href=#transformer>Transformer</a></h3><p><code>postgres-cluster-image-transformer.yaml</code><pre><code class=language-yaml>images:
- path: spec/image
 kind: PostgresCluster
</code></pre><h3 id=kustomizationyaml><a href=#kustomizationyaml>Kustomization.yaml</a></h3><p><code>kustomization.yaml</code><h4 id=before><a href=#before>Before</a></h4><pre><code class=language-yaml>resources:
  - postgres-self-test-config.yaml
  - postgres.yaml
</code></pre><h4 id=after><a href=#after>After</a></h4><pre><code class=language-yaml>configurations:
- postgres-cluster-image-transformer.yaml
images:
- name: bobpachcrunchy/crunchy-postgres
 newTag: ubi8-15.1-5.3.0-1
resources:
- postgres-self-test-config.yaml
- postgres.yaml
</code></pre><h3 id=postgres-cluster><a href=#postgres-cluster>Postgres Cluster</a></h3><p><code>postgres.yaml</code><h4 id=before-1><a href=#before-1>Before</a></h4><pre><code class=language-yaml>spec:
  image: bobpachcrunchy/crunchy-postgres:ubi8-15.1-5.3.0-1
</code></pre><h4 id=after-1><a href=#after-1>After</a></h4><pre><code class=language-yaml>spec:
  image: bobpachcrunchy/crunchy-postgres
</code></pre><p>Check these changes into your git repo. They will be required before updating the Postgres image in the registry.<h2 id=deploy-the-clusters><a href=#deploy-the-clusters>Deploy the clusters</a></h2><p>If you don't already have your Postgres clusters up and running, sync the postgres-dev Argo CD application. This will deploy the Postgres cluster to the postgres-dev namespace and will run the self test which will sync the Postgres cluster to the postgres-qa namespace.<p>Verify both clusters deployed.<pre><code class=language-shell>$ kubectl -n postgres-dev get pods
NAME                     READY  STATUS    RESTARTS  AGE
hippo-backup-5vvr-h2zmd  0/1    Completed   0       3m41s
hippo-pgha1-7lng-0       5/5    Running     0       4m2s
hippo-pgha1-g8xx-0       5/5    Running     0       4m3s
hippo-pgha1-nnrm-0       5/5    Running     0       4m2s
hippo-repo-host-0        2/2    Running     0       4m2s

$ kustomize % kubectl -n postgres-qa get pods
NAME                     READY  STATUS    RESTARTS  AGE
hippo-backup-r4p8-z4689  0/1    Completed   0       3m32s
hippo-pgha1-4992-0       5/5    Running     0       3m54s
hippo-pgha1-4cmv-0       5/5    Running     0       3m54s
hippo-pgha1-mzfm-0       5/5    Running     0       3m54s
hippo-repo-host-0        2/2    Running     0       3m54s
</code></pre><p>Both Postgres clusters are up and running. Lets take a look at the Postgres image version we deployed for each cluster. We will describe the stateful sets for each namespace. Notice the crunchy-postgres image tag is currently ubi8-5.3.0-1. This is our starting image.<p>Note: Results shown below have been truncated for readability.<pre><code class=language-shell>$ kubectl get -n postgres-dev sts -o wide | grep crunchy-postgres
hippo-pgha1-7lng  1/1   7m2s  bobpachcrunchy/crunchy-postgres:ubi8-5.3.0-1
hippo-pgha1-g8xx  1/1   7m3s  bobpachcrunchy/crunchy-postgres:ubi8-5.3.0-1
hippo-pgha1-nnrm  1/1   7m3s  bobpachcrunchy/crunchy-postgres:ubi8-5.3.0-1

$ kubectl get -n postgres-qa sts -o wide | grep crunchy-postgres
hippo-pgha1-4992  1/1   7m1s  bobpachcrunchy/crunchy-postgres:ubi8-5.3.0-1
hippo-pgha1-4cmv  1/1   7m1s  bobpachcrunchy/crunchy-postgres:ubi8-5.3.0-1
hippo-pgha1-mzfm  1/1   7m1s  bobpachcrunchy/crunchy-postgres:ubi8-5.3.0-1
</code></pre><h2 id=time-to-automate-updates><a href=#time-to-automate-updates>Time to Automate Updates</a></h2><p>We have completed all of the prep work. Now it's time to automate updates. We need to make one more change to the postgres-dev application in Argo CD. We will edit the application and enable auto-sync.<ul><li>Click on applications in the left panel of the Argo CD UI.<li>Click on the postgres-dev application<li>Click the App Details button in the top panel.<li>Click the Enable Auto-Sync button in the Sync Policy Pane of the panel.<li>Click OK on the confirmation dialogue.</ul><p>We are ready to push an updated image into our private registry. Here is my registry before I push the updated image: <img alt=registry_before loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/588b55d4-7529-435c-649e-3840e951ce00/public><p>Ensure you have Docker running. We will pull the new image, tag it, and push it into our private registry.<pre><code class=language-shell>docker pull registry.crunchydata.com/crunchydata/crunchy-postgres:ubi8-15.3-5.3.2-1
docker tag registry.crunchydata.com/crunchydata/crunchy-postgres:ubi8-15.3-5.3.2-1 bobpachcrunchy/crunchy-postgres:ubi8-15.3-5.3.2-1
docker push bobpachcrunchy/crunchy-postgres:ubi8-15.3-5.3.2-1
</code></pre><p>I now see my new image in my private repo: <img alt=registry_after loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/588b55d4-7529-435c-649e-3840e951ce00/public><p>Argo CD Image Updater is monitoring my Docker registry. It see's that I have a new image and it connects to my git repo to update the image tag in my kustomization.yaml file.<p>A quick look at the argo-cd-image-update logs shows:<pre><code class=language-text>time="2023-08-03T19:36:23Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=0"                │
time="2023-08-03T19:38:23Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"                          │
time="2023-08-03T19:38:23Z" level=info msg="Setting new image to bobpachcrunchy/crunchy-postgres:ubi8-15.3-5.3.2-1" alias=postgres application=postgres-dev image_name=bobpac │
time="2023-08-03T19:38:23Z" level=info msg="Successfully updated image 'bobpachcrunchy/crunchy-postgres:ubi8-15.1-5.3.0-1' to 'bobpachcrunchy/crunchy-postgres:ubi8-15.3-5.3. │
time="2023-08-03T19:38:23Z" level=info msg="Committing 1 parameter update(s) for application postgres-dev" application=postgres-dev
</code></pre><p>A look in git shows a commit message of: "build: automatic update of postgres-dev". The kustomization.yaml file now has the updated image tag.<pre><code class=language-yaml>configurations:
- postgres-cluster-image-transformer.yaml
images:
- name: bobpachcrunchy/crunchy-postgres
 newTag: ubi8-15.3-5.3.2-1
resources:
- postgres-self-test-config.yaml
- postgres.yaml
</code></pre><p>The postgres-dev Argo CD app is set to auto-sync. By default auto-sync will fire every 3 minutes. When it does fire, it will see that it is out of sync with the git repo and re-apply the manifest. The Postgres pods go into a rolling restart process. Each replica pod will be taken down one at a time and re-initialized with the new image. After all replica pods are updated a failover happens to elect an updated replica to the primary pod role. The former primary is then brought down and re-initialized as a replica with the new image. At this point, all Postgres pods in the cluster are running the new image with the only downtime being the few seconds for failover to happen.<p>The self-test container ran after the replica was promoted to primary. It then calls the sync command on the postgres-qa Argo CD app and that Postgres cluster also gets its images updated in the same manner.<p>Both Postgres clusters have now been updated. Let's take a look at the Postgres image version we updated for each cluster. We will describe the stateful sets for each namespace. Notice the crunchy-postgres image tag is now ubi8-15.3-5.3.2-1. This is our updated image.<p>Note: Results shown below have been truncated for readability.<pre><code class=language-shell>$ kubectl get -n postgres-dev sts -o wide | grep crunchy-postgres
hippo-pgha1-svkv  1/1   17m  bobpachcrunchy/crunchy-postgres:ubi8-15.3-5.3.2-1
hippo-pgha1-x7qf  1/1   17m  bobpachcrunchy/crunchy-postgres:ubi8-15.3-5.3.2-1
hippo-pgha1-zgcq  1/1   17m  bobpachcrunchy/crunchy-postgres:ubi8-15.3-5.3.2-1

$ kubectl get -n postgres-qa sts -o wide | grep crunchy-postgres
hippo-pgha1-fw9r  1/1   18m  bobpachcrunchy/crunchy-postgres:ubi8-15.3-5.3.2-1
hippo-pgha1-r9zk  1/1   18m  bobpachcrunchy/crunchy-postgres:ubi8-15.3-5.3.2-1
hippo-pgha1-tf22  1/1   18m  bobpachcrunchy/crunchy-postgres:ubi8-15.3-5.3.2-1
</code></pre><h2 id=summary><a href=#summary>Summary</a></h2><p>Using Argo CD, Git, Docker Hub, and the self-test container we were able to automatically deploy, test, and promote a new Postgres image to Postgres clusters in Dev and QA namespaces by doing nothing more than pushing an updated image into a Docker registry that argocd-image-updater was monitoring. Many organizations will use additional gating mechanisms like manual user acceptance testing and management sign-off before allowing promotions to Prod. The processes outlined in this blog can be modified to align with your specific prod gating requirements.<p>Test automation is a critical component of any automated CI/CD pipeline. Today's test harnesses are designed to support concurrency, use case permutations and process iterations. The more testing you automate the more rapidly you can move packages through you pipeline. Using CI/CD automation facilitates a decrease in time to market with your new container images without sacrificing quality or reliability.<p>A final view of the entire possible workflow:<p><img alt="full workflow"loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/3ad7c6c3-c160-4d22-55c8-8cdb1523b200/public> ]]></content:encoded>
<category><![CDATA[ Kubernetes ]]></category>
<author><![CDATA[ Bob.Pacheco@crunchydata.com (Bob Pacheco) ]]></author>
<dc:creator><![CDATA[ Bob Pacheco ]]></dc:creator>
<guid isPermalink="false">781bd608c1ed7b1c7c1fed0987cf8257c4dbf0f0090e581f792bfd1f9c4e1fd9</guid>
<pubDate>Fri, 18 Aug 2023 09:00:00 EDT</pubDate>
<dc:date>2023-08-18T13:00:00.000Z</dc:date>
<atom:updated>2023-08-18T13:00:00.000Z</atom:updated></item>
<item><title><![CDATA[ CI/CD with Crunchy Postgres for Kubernetes and Argo ]]></title>
<link>https://www.crunchydata.com/blog/ci-cd-with-crunchy-postgres-for-kubernetes-and-argo</link>
<description><![CDATA[ CI/CD tools can help you deploy, test, and promote databases. Bob has a sample set up of using ArgoCD with Crunchy Postgres for Kubernetes. He even provides a really nice sidecar testing container for you to use.  ]]></description>
<content:encoded><![CDATA[ <p>Continuous Integration / Continuous Delivery (CI/CD) is an automated approach in which incremental code changes are made, built, tested and delivered. Organizations want to get their software solutions to market as quickly as possible without sacrificing quality or stability. While CI/CD is often associated with application code, it can also be beneficial for managing changes to PostgreSQL database clusters.<p>GitOps plays an important part in enabling CI/CD. If you are unfamiliar with GitOps, I recommend starting with <a href=https://www.crunchydata.com/blog/postgres-gitops-with-argo-and-kubernetes>my previous post on Postgres GitOps with Argo and Kubernetes</a>.<p>Today I have a working sample to walk through for implementing basic CI/CD using Crunchy Postgres for Kubernetes, ArgoCD, and a self test container. In the following steps we will:<ul><li>Deploy a Postgres cluster to a developer namespace<li>Run a series of tests on the deployed cluster<li>Once the tests pass we will automatically deploy the same postgres cluster to a QA namespace.</ul><h3 id=prerequisites><a href=#prerequisites>Prerequisites</a></h3><ul><li><a href=https://access.crunchydata.com/documentation/postgres-operator/latest/quickstart/>Crunchy Postgres for Kubernetes</a> v5.3 or later.<li>The sidecar functionality should be enabled. To enable the sidecar functionality you will need to add the following to the <code>Deployment.spec.template.spec.containers.env</code> section of the <code>manager.yaml</code> file located in the <code>postgres-operator-examples/kustomize/install/manager</code> directory.</ul><pre><code class=language-yaml>- name: PGO_FEATURE_GATES
  value: 'InstanceSidecars=true'
</code></pre><ul><li><a href=https://argo-cd.readthedocs.io/en/release-2.0/getting_started/>ArgoCD</a> v 2.6 or later deployed in the Kubernetes cluster.<li>A private container registry containing the images you want to deploy. Most organizations will pull images, tag them and then upload them into their private registries. For this blog I am using a private registry for all images except the self test. That image is in a public repo in my docker registry.<li>A git repository containing the Crunchy Postgres for Kubernetes manifest to be deployed. Here's a sample manifest you can use or you can fork <a href=https://github.com/bobpach/Postgres-CI-CD>my git repository</a>.</ul><details><summary>- kustomization.yaml</summary><pre><code class=language-yaml>resources:
  # - argocd-token.yaml
  - hippo-self-test-config.yaml
  - postgres.yaml
</code></pre></details> <details><summary>- postgres.yaml</summary><pre><code class=language-yaml>apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: hippo
spec:
  image: &#60your_registry>/crunchy-postgres:ubi8-15.2-5.3.1-2
  imagePullSecrets:
    - name: privatereg
  postgresVersion: 15
  shutdown: false
  users:
    - name: hippo
      options: 'SUPERUSER'
  instances:
    - name: pgha1
      replicas: 3
      containers:
        - name: selftest
          image: bobpachcrunchy/postgres-self-test:1.0.0
          imagePullPolicy: IfNotPresent
          env:
            - name: ARGOCD_APP_NAME
              valueFrom:
                configMapKeyRef:
                  name: postgres-self-test-config
                  key: auto-promote-argocd-app-name
            - name: ARGOCD_NAMESPACE
              valueFrom:
                configMapKeyRef:
                  name: postgres-self-test-config
                  key: argocd-namespace
            - name: AUTO_PROMOTE
              valueFrom:
                configMapKeyRef:
                  name: postgres-self-test-config
                  key: auto-promote
            - name: ARGOCD_SERVICE_ADDRESS
              valueFrom:
                configMapKeyRef:
                  name: postgres-self-test-config
                  key: argocd-service-address
            - name: ARGOCD_TOKEN
              valueFrom:
                secretKeyRef:
                  key: token
                  name: argocd-token
            - name: ARGOCD_VERIFY_TLS
              valueFrom:
                configMapKeyRef:
                  name: postgres-self-test-config
                  key: argocd-verify-tls
            - name: DB_USER
              valueFrom:
                configMapKeyRef:
                  name: postgres-self-test-config
                  key: db-user
            - name: DB_USER_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: hippo-pguser-hippo
            - name: CLUSTER_NAME
              valueFrom:
                configMapKeyRef:
                  name: postgres-self-test-config
                  key: cluster-name
            - name: LOG_LEVEL
              valueFrom:
                configMapKeyRef:
                  name: postgres-self-test-config
                  key: log-level
            - name: LOG_PATH
              valueFrom:
                configMapKeyRef:
                  name: postgres-self-test-config
                  key: log-path
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: POSTGRES_CONN_ATTEMPTS
              valueFrom:
                configMapKeyRef:
                  name: postgres-self-test-config
                  key: postgres-conn-attempts
            - name: POSTGRES_CONN_INTERVAL
              valueFrom:
                configMapKeyRef:
                  name: postgres-self-test-config
                  key: postgres-conn-interval
            - name: SERVICE_PORT
              valueFrom:
                configMapKeyRef:
                  name: postgres-self-test-config
                  key: service-port
            - name: SSLMODE
              valueFrom:
                configMapKeyRef:
                  name: postgres-self-test-config
                  key: sslmode
          volumeMounts:
            - name: postgres-data
              readOnly: false
              mountPath: /pgdata
      dataVolumeClaimSpec:
        accessModes:
          - 'ReadWriteOnce'
        resources:
          requests:
            storage: 1Gi
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels:
                    postgres-operator.crunchydata.com/cluster: hippo
                    postgres-operator.crunchydata.com/instance-set: pgha1
  backups:
    pgbackrest:
      image: &#60your_registry>/crunchy-pgbackrest:ubi8-5.3.1-2
      repos:
        - name: repo1
          volume:
            volumeClaimSpec:
              accessModes:
                - 'ReadWriteOnce'
              resources:
                requests:
                  storage: 1Gi
</code></pre></details>
</br><p><strong>Note:</strong> For this blog the Postgres cluster name is "hippo". You should substitute this name and any other relevant values with your proper information.<h2 id=self-test-container><a href=#self-test-container>Self Test Container</a></h2><p>I have created a self test container for Crunchy Postgres for Kubernetes. This will be deployed as a sidecar in each Postgres pod. It runs read, write and delete tests in the cluster and confirms that replication is working as expected across all Postgres pods. If the tests pass it will synch an ArgoCD application resulting in the promotion of the Postgres cluster to another namespace. The sidecar uses a configmap to manage self test behavior and ArgoCD application sync. More information about the self test container and its configuration can be found <a href=https://github.com/CrunchyData/postgres-ci-cd-demo/tree/main/Self-Test-Container>in github</a>.</p><details><summary>- postgres-self-test-config.yaml</summary><pre><code class=language-yaml>apiVersion: v1
data:
  argocd-namespace: argocd
  argocd-service-address: &#60ip address of the argocd service>
  argocd-verify-tls: 'false'
  auto-promote: 'true'
  auto-promote-argocd-app-name: postgres-qa
  db-user: hippo
  cluster-name: hippo
  log-level: info
  log-path: /pgdata
  postgres-conn-attempts: '12'
  postgres-conn-interval: '5'
  service-port: '5432'
  sslmode: require
kind: ConfigMap
metadata:
  labels:
    vendor: crunchydata
    postgres-operator.crunchydata.com/cluster: hippo
  name: postgres-self-test-config
</code></pre></details><h2 id=argocd><a href=#argocd>ArgoCD</a></h2><p>You will need an ArgoCD token to connect to the server to synch the target application after the tests pass. To get an ArgoCD token you will need to create a repository, project, role and policy. This can be done through the UI or CLI. For this blog we will use the UI. See ArgoCD documentation for CLI installation and commands.<h3 id=create-an-argocd-repository><a href=#create-an-argocd-repository>Create an ArgoCD Repository</a></h3><p>To add your git repository to Argo CD click on ‘Settings’ in the navigation bar on the left.<p>Click on ‘Repositories’ and the ‘Connect Repo’ buttons. Provide the necessary information and click ‘Save’. You should now have a connected repo. If not, please review your settings and try again.<h3 id=create-an-argocd-project><a href=#create-an-argocd-project>Create an ArgoCD Project</a></h3><details>
<summary><p><strong>Create an ArgoCD Project with the following properties:</strong><ul><li>Project Name: cicd-project<li>Description: A project used for the CI/CD pipeline.<li>Source Repositories: &#60 the git repository you registered with argocd ><li>Destinations:<ul><li>Server: <a href=https://kubernetes.default.svc>https://kubernetes.default.svc</a><li>Name: in-cluster<li>Namespace: *</ul></ul></summary><p><img alt=ci-cd-project loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/9bb84f06-9618-4f3a-9113-a32e7f08fa00/public></p></details> <details>
<summary><p><strong>Click on the Roles tab and click the Add Role button. Create a role and policy with the following properties:</strong><ul><li>Role Name: cicd-sync<li>Description: Project role used to synch apps from the CI/CD pipeline.<li>Action: sync<li>Application: cicd-project/*<li>Permission: allow</ul></summary><p><img alt=cicd-project-role loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/908a535d-087e-43a1-9a69-58ad84dbe000/public></p></details> <details>
<summary><p><strong>Click on the role name you just created and then click the Create button in the JWT Tokens section of the sidebar.</strong> Click OK when prompted. Copy and save the New Token in a safe place.</p></summary><p><img alt=cicd-project-role-details loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/ed39fa4c-92e4-4c39-0dd6-b742fa28ce00/public></p></details><p><strong>Base 64 encode the New Token.</strong><pre><code class=language-shell>echo &#60 your new token > | base64
</code></pre><p>Copy the base64 encoded new token and paste it into the argocd-token.yaml file</p><details><summary>- argocd-token.yaml</summary><pre><code class=language-yaml>apiVersion: v1
data:
  token: &#60your base64 encoded argocd token>
kind: Secret
metadata:
  name: argocd-token
type: Opaque
</code></pre></details><p>For security reasons, you may choose not to store your Argocd token in git. If that is the case you can create the secret directly in your two namespaces.<pre><code class=language-shell>$ kubectl apply -n postgres-dev -f argocd-token.yaml
secret/argocd-token created
$ kubectl apply -n postgres-qa -f argocd-token.yaml
secret/argocd-token created
</code></pre><p>Otherwise, you can uncomment the # - argocd-token.yaml line in the kustomization.yaml file and it will get created when the cluster gets created.<h3 id=create-the-argocd-applications><a href=#create-the-argocd-applications>Create the ArgoCD Applications</a></h3><p>We will create two Argocd applications. Create the first one with the following properties:<ul><li>General<ul><li>Application Name: <strong>postgres-dev</strong><li>Project Name: cicd-project</ul><li>Source<ul><li>Repository URL: &#60 the git repository you registered with argocd ><li>Path: &#60 the path to the kustomization file in the git repo ></ul><li>Destination:<ul><li>Cluster URL: <code>https://kubernetes.default.svc</code><li>Namespace: <strong>postgres-dev</strong></ul></ul><p>Create the second application with the same properties as the first one with two exceptions:<ul><li>General<ul><li>Application Name: <strong>postgres-qa</strong></ul><li>Destination:<ul><li>Namespace: <strong>postgres-qa</strong></ul></ul><p><img alt=applications loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/1f111c4e-dce9-46f3-dcd2-8e29294c6000/public><h2 id=deploy-the-crunchy-postgres-for-kubernetes-cluster><a href=#deploy-the-crunchy-postgres-for-kubernetes-cluster>Deploy the Crunchy Postgres for Kubernetes Cluster</a></h2><p><strong>Note:</strong> Ensure that all yaml files have been checked into the git repo that you assigned to the cicd-project in ArgoCD. These files should have all of your edits. They will be used by ArgoCD to deploy into your postgres-dev and postgres-qa namespaces.<p>Click the synch button in the postgres-dev Argocd application then click the synchronize button in the right side panel. The postgres cluster will get deployed into the postgres-dev namespace.<p>Let's look at the pods in our postgres-dev namespace:<pre><code class=language-shell>$ kubectl get pods -n postgres-dev
NAME                      READY   STATUS      RESTARTS   AGE
hippo-backup-s5pz-bmjsd   0/1     Completed   0          2m5s
hippo-pgha1-2zjd-0        5/5     Running     0          2m29s
hippo-pgha1-9kf7-0        5/5     Running     0          2m29s
hippo-pgha1-mrhq-0        5/5     Running     0          2m29s
hippo-repo-host-0         2/2     Running     0          2m29s
</code></pre><p>Take a look at the self test container logs in the primary Postgres pod in the postgres-dev namespace to see if our tests passed.<pre><code class=language-shell>$ kubectl logs $(kubectl get pod -l postgres-operator.crunchydata.com/role=master -o name -n postgres-dev) -n postgres-dev -c selftest
2023-05-24 14:44:24,961 - self_test -           INFO - ******* STARTING NEW TEST RUN *******
2023-05-24 14:44:30,024 - self_test -           INFO - PostgreSQL database version:
2023-05-24 14:44:30,025 - self_test -           INFO - ('PostgreSQL 15.1 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15), 64-bit',)
2023-05-24 14:44:30,032 - self_test -           INFO - Creating test database
2023-05-24 14:44:30,186 - self_test -           INFO - Assigning test_db privileges to test_user
2023-05-24 14:44:30,210 - self_test -           INFO - Creating test_schema in test_db
2023-05-24 14:44:30,404 - self_test -           INFO - Creating test_table with data in test_schema
2023-05-24 14:44:30,417 - self_test -           INFO - Validating DBConnectionType.PRIMARY_SERVICE Data: Expecting 1000 Rows
2023-05-24 14:44:30,418 - self_test -           INFO - *** DBConnectionType.PRIMARY_SERVICE Validation Succeeded! ***
2023-05-24 14:44:40,458 - self_test -           INFO - Validating DBConnectionType.REPLICA_SERVICE Data: Expecting 1000 Rows
2023-05-24 14:44:40,460 - self_test -           INFO - *** DBConnectionType.REPLICA_SERVICE Validation Succeeded! ***
2023-05-24 14:44:40,472 - self_test -           INFO - Validating DBConnectionType.REPLICA_POD Data for pod hippo-pgha1-2zjd-0: Expecting 1000 Rows
2023-05-24 14:44:40,473 - self_test -           INFO - *** DBConnectionType.REPLICA_POD Validation Succeeded for pod hippo-pgha1-2zjd-0! ***
2023-05-24 14:44:40,481 - self_test -           INFO - Validating DBConnectionType.REPLICA_POD Data for pod hippo-pgha1-9kf7-0: Expecting 1000 Rows
2023-05-24 14:44:40,482 - self_test -           INFO - *** DBConnectionType.REPLICA_POD Validation Succeeded for pod hippo-pgha1-9kf7-0! ***
2023-05-24 14:44:40,663 - self_test -           INFO - Successfully synched the postgres-qa ArgoCD application.
2023-05-24 14:44:40,663 - self_test -           INFO - ******* SUCCESS: ALL TESTS PASSED *******
2023-05-24 14:44:40,664 - self_test -           INFO - Dropping test_table
2023-05-24 14:44:40,667 - self_test -           INFO - Dropping test_schema
2023-05-24 14:44:40,670 - self_test -           INFO - Dropping test_db
2023-05-24 14:44:40,697 - self_test -           INFO - Dropping test_user
</code></pre><p>Now let's look at the pods in our postgres-qa namespace:<pre><code class=language-shell>$ kubectl get pods -n postgres-qa
NAME                      READY   STATUS      RESTARTS   AGE
hippo-backup-wxrx-rhzzh   0/1     Completed   0          63s
hippo-pgha1-7x55-0        5/5     Running     0          81s
hippo-pgha1-jdxw-0        5/5     Running     0          81s
hippo-pgha1-lp89-0        5/5     Running     0          81s
hippo-repo-host-0         2/2     Running     0          81s
</code></pre><p>Lastly, look at the two ArgoCD applications. They are now both marked as Synched.<p><img alt="synched apps"loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/a652221f-7594-45f1-9586-c31c10a8ab00/public><h2 id=summary><a href=#summary>Summary</a></h2><p>We were able to deploy, test and promote a Crunchy Postgres for Kubernetes cluster using git and ArgoCD. The declarative nature of the manifest combined with the power of GitOps and ArgoCD makes creating a CI/CD pipeline easier than ever. This blog only covers a segment of a full end to end pipeline. In my next blog we will look at how to apply new images to an existing application using ArgoCD Image Updater. Stay tuned for part 2. ]]></content:encoded>
<category><![CDATA[ Kubernetes ]]></category>
<author><![CDATA[ Bob.Pacheco@crunchydata.com (Bob Pacheco) ]]></author>
<dc:creator><![CDATA[ Bob Pacheco ]]></dc:creator>
<guid isPermalink="false">c12fe69cf949f08dc7d2be3c3a2121c14606cce31508d3a147548fcd917f1995</guid>
<pubDate>Wed, 07 Jun 2023 09:00:00 EDT</pubDate>
<dc:date>2023-06-07T13:00:00.000Z</dc:date>
<atom:updated>2023-06-07T13:00:00.000Z</atom:updated></item>
<item><title><![CDATA[ Postgres GitOps with Argo and Kubernetes ]]></title>
<link>https://www.crunchydata.com/blog/postgres-gitops-with-argo-and-kubernetes</link>
<description><![CDATA[ Curious about using Argo for application and database maintenance? Bob takes us through how to set up Argo with Crunchy Postgres for Kubernetes. The end results are some easy and simple GitOps for everyday tasks. ]]></description>
<content:encoded><![CDATA[ <p>Postgres clusters should be updated regularly and have routine maintenance. This regular maintenance is often referred to as “Day 2 operations” and can include a wide variety of tasks like restarting services, resetting passwords, or updating versions. Performing Day 2 operations can be complex and time consuming, especially if you are supporting a large number of Postgres clusters.<p>With the adoption of GitOps and a little help from continuous delivery tools like <a href=https://argo-cd.readthedocs.io/en/stable/>Argo CD</a> you can simplify your day 2 operations. The declarative nature of <a href=https://www.crunchydata.com/products/crunchy-postgresql-for-kubernetes>Crunchy Postgres for Kubernetes</a> makes it a perfect candidate to integrate into the GitOps world. Let’s take a look at how GitOps and Argo CD can help perform some of these routine Postgres tasks.<p>The examples below assume that you have installed the <a href=https://www.crunchydata.com/developers/get-started/postgres-operator>Postgres Operator</a> v5.2 or later in the same Kubernetes cluster that you will be deploying the PostgreSQL clusters in.<h2 id=the-git-in-gitops><a href=#the-git-in-gitops>The “Git” in GitOps</a></h2><p>In order to perform GitOps operations you must check-in your infrastructure files to Git. In my GitHub repo I have a sample set of <a href=https://github.com/bobpach/blogs/tree/master/PostgresGitOps/gitops>Kustomize files in a public repo</a> that separates the files into admin, base and overlay directories. This structure helps isolate files so they can be independently run via <a href=https://kustomize.io/>Kustomize</a> in Argo CD. Feel free to fork and clone the repo if you would like to use these files to follow along.<p>In GitOps, Git is used as the ‘Single Source of Truth’ to manage all infrastructure files. The files in my GitHub repo manage artifacts to deploy the Postgres clusters and perform declarative day 2 operations on them.<p>You can copy and paste the example files into your own files and check them in to your git repo. Take care to replace the <code>&#60cluster_name></code> place holder with your own cluster name and use the same value in all of the example files.<h2 id=argo-cd><a href=#argo-cd>Argo CD</a></h2><p>Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. It provides a UI and CLI that makes it easy to deploy and manage your applications running on Kubernetes. To install Argo CD in your cluster simply run the following:<pre><code class=language-shell>kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
</code></pre><p>Additional details can be found in their <a href=https://argo-cd.readthedocs.io/en/stable/getting_started/>getting started guide</a>.<p>To access the Argo CD UI I have changed the argocd-sever service from ClusterIP to LoadBalancer. This provides an external IP that I can access from my browser. You could also use a port forward or an ingress.<pre><code class=language-shell>kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'
</code></pre><p>Once you have launched the Argo CD UI you can login. The initial password for the <strong><em>admin</em></strong> account can be retrieved from the argocd-initial-admin-secret.<pre><code class=language-shell>kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
</code></pre><h3 id=configuring-argo-cd><a href=#configuring-argo-cd>Configuring ARGO CD</a></h3><p>Now that you have logged in you are ready to start configuring Argo CD to deploy and manage your Postgres cluster on Kubernetes.<h3 id=add-the-git-repository><a href=#add-the-git-repository>Add the Git Repository</a></h3><p>You need to add your git repository to Argo CD. Click on ‘Settings’ in the navigation bar on the left.<p>Now click on ‘Repositories’ and the ‘Connect Repo’ buttons. Provide the necessary information and click ‘Save’. You should now have a connected repo. If not, please review your settings and try again.<p><img alt=argo-conntect-repo.png loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/9519b4cb-4730-46d0-23ff-454b86a03e00/original><h3 id=create-the-projects><a href=#create-the-projects>Create the Projects</a></h3><p>Argo CD uses projects to logically group applications. I created three projects:<ul><li><p>deploy<li><p>reset-password<li><p>start-stop</ul><p>To create a project click on ‘Settings’ in the navigation bar on the left. Click on ‘Project’ and then the ‘New Project’ button. Provide a project name and description in the panel on the right and click create. Add your repo to ‘Source Repositories’ and add your Kubernetes cluster to ‘Destinations’ as seen below. Repeat the steps for each project you want to create.<p><img alt=argo-deploy-app-list.png loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/0297b9d0-19ab-412a-61c7-1ce864446f00/original><p>For this blog I deployed Argo CD in the same Kubernetes cluster that I am deploying Postgres into. You can refer to the <a href=https://argo-cd.readthedocs.io/en/stable/>Argo CD Documentation</a> for other deployment options.<h2 id=creating-applications><a href=#creating-applications>Creating Applications</a></h2><p>Argo CD uses applications to perform declarative operations on its configured Kubernetes cluster. We will create several applications to deploy Postgres clusters and perform day 2 operations across three namespaces.<p><strong><em>Operations</em></strong><ul><li>Deploy Postgres<li>Start Postgres<li>Stop Postgres<li>Reset Password<li>Upgrade Postgres - Minor Version</ul><p>Create the namespaces in the same Kubernetes cluster that you deployed Argo CD into.<pre><code class=language-shell>kubectl create namespace gitopsdemo-dev
kubectl create namespace gitopsdemo-qa
kubectl create namespace gitopsdemo-prod
</code></pre><h3 id=deploy><a href=#deploy>Deploy</a></h3><p>Let’s start with the deploy application. Click on ‘Applications’ in the navigation bar on the left. Click on the ‘New App’ button. We will create ‘deploy-dev’ application. Notice that I am assigning this to the deploy project and the gitopsdemo-dev namespace. I have also added my github repo that I previously registered and I assigned the correct repo path to where the kustomization.yaml file resides.<p>Create two additional applications for the two remaining namespaces naming them accordingly: deploy-qa, deploy-prod. When you are done your application list for the deploy project will look like this:<p><img alt=argo-deploy-projects.png loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/0297b9d0-19ab-412a-61c7-1ce864446f00/original><h3 id=start--stop><a href=#start--stop>Start / Stop</a></h3><p>Now let’s create the applications for starting and stopping the Postgres cluster in our start-stop project for all three namespaces. Ensure that you select the correct namespace and the correct GitHub repo path for each application. Your completed application list for the start-stop project will look like this:<p><img alt=start-stop-app-list.png loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/e99361b9-53a5-436a-eee2-0ac717883f00/original><h3 id=reset-password><a href=#reset-password>Reset Password</a></h3><p>The <dfn>Postgres Operator</dfn> (<abbr>PGO</abbr>) in CPK will automatically recreate a password in a secret it manages if that password is blanked out (""). We will use this feature in our next set of applications. Create a reset-password application in the reset-password project for each namespace. Again, be mindful to select the proper namespace and GitHub repo path for each application. These applications will be configured like the previous with one exception. Check the ‘Skip Schema Validation’ checkbox. Kubernetes does not like empty passwords in secrets. However, the password will be immediately regenerated with a new random password by the operator.<p>Your completed application list for the reset-password project will look like this:<p><img alt=argo-reset-password.png loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/4ea618d9-d1cb-4d64-f5c4-b0706126a800/original><h2 id=gitops-in-action><a href=#gitops-in-action>GitOps in Action</a></h2><p>You have taken the time to configure Argo CD and set up all of the applications you require to deploy Postgres clusters and perform day 2 operations. Now you can reap the benefits of your efforts.<h3 id=deploy-1><a href=#deploy-1>Deploy</a></h3><p>Click on the ‘Sync’ button in the deploy-dev application in the deploy project:<p><img alt=argo5.png loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/024b0a70-b17d-4c0d-9dee-77209884d500/public><p>Click on the ‘Synchronize’ button on the right panel. Now do the same for the deploy-qa and deploy-prod applications. You will notice that each application is now marked as "Synched’.<p>I can see that Postgres was deployed and backed up when I request the pod list from the three namespaces:<pre><code class=language-shell>$ kubectl -n gitopsdemo-dev get pods
NAME                      READY   STATUS      RESTARTS   AGE
hippo-backup-vh2n-dpw8f   0/1     Completed   0          4m36s
hippo-pgdb-frxt-0         4/4     Running     0          5m30s
hippo-repo-host-0         2/2     Running     0          5m30s

$ kubectl -n gitopsdemo-qa get pods
NAME                      READY   STATUS      RESTARTS   AGE
hippo-backup-s8cw-85kn9   0/1     Completed   0          57s
hippo-pgdb-dfrr-0         5/5     Running     0          75s
hippo-pgdb-n6fh-0         5/5     Running     0          75s
hippo-repo-host-0         2/2     Running     0          75s

$ kubectl -n gitopsdemo-prod get pods
NAME                               READY   STATUS      RESTARTS   AGE
hippo-backup-qwzb-gkb75            0/1     Completed   0          51s
hippo-pgbouncer-7444dc559c-bsjmw   2/2     Running     0          67s
hippo-pgbouncer-7444dc559c-rk992   2/2     Running     0          67s
hippo-pgdb-b5hw-0                  5/5     Running     0          68s
hippo-pgdb-knvc-0                  5/5     Running     0          68s
hippo-repo-host-0                  2/2     Running     0          67s
</code></pre><p>Notice the Postgres cluster differences between namespaces. The gitopsdemo-dev namespace has one Postgres pod with four containers, a repo host and a completed backup job. The gitopsdemo-qa namespace has two Postgres pods with five containers each, a repo host and a completed backup job. The gitopsdemo-prod namespace has two pgBouncer pods in addition to the two postgres pods, repo host and completed backup job.<p>The use of overlays allows you to standardize your base configuration and then “overlay” that configuration with specific changes. Take a closer look at the base and overlay yaml files we are using here to see what else has been changed per instance.<h3 id=start--stop-1><a href=#start--stop-1>Start / Stop</a></h3><p>The declarative nature of Crunchy PostgreSQL for Kubernetes works perfectly with Argo CD to simplify Postgres Cluster deployments the GitOps way. Let’s stop the dev Postgres cluster. Select the stop-dev application from the start-stop project. Click on ‘Synch’ and then click on ‘Synchronize’ in the right panel. You will notice that the application is now marked as ‘Synched’ and the dev pods have been stopped. The backup job is still listed as completed. It was not stopped because as a completed job it was not running.<pre><code class=language-shell>$ kubectl -n gitopsdemo-dev get pods
NAME                      READY   STATUS      RESTARTS   AGE
hippo-backup-vh2n-dpw8f   0/1     Completed   0          58m
</code></pre><p>Now let’s start the dev Postgres cluster. Select the start-dev application from the start-stop project. Click on ‘Synch’ and then click on ‘Synchronize’ in the right panel. You will notice that the application is now marked as ‘Synched’ and the dev pods have been started. You will also notice that the stop-dev job is now marked as ‘OutOfSynch’.<pre><code class=language-shell>$ kubectl -n gitopsdemo-dev get pods
NAME                      READY   STATUS      RESTARTS   AGE
hippo-backup-vh2n-dpw8f   0/1     Completed   0          61m
hippo-pgdb-frxt-0         4/4     Running     0          18s
hippo-repo-host-0         2/2     Running     0          18s
</code></pre><p>Try this with the qa and prod applications as well. It becomes very easy to start or stop a Postgres cluster when needed using GitOps.<h3 id=reset-password-1><a href=#reset-password-1>Reset Password</a></h3><p>Your security team has decided it is time to reset the password for the initial user created at deploy time. First determine what the current password is so we can validate that the change took place.<pre><code class=language-shell>kubectl -n gitopsdemo-dev get secret &#60cluster_name-pguser-cluster_name> -o jsonpath="{.data.password}" | base64 -d; echo
</code></pre><p>Make a note of that password. Select the reset-password-dev application from the reset-password project. Click on ‘Synch’ and then click on ‘Synchronize’ in the right panel. In this case the status of the application stays listed as ‘OutOfSynch’ because the operator immediately assigned a new randomly generated password after it was blanked out. Let’s run the same command again.<pre><code class=language-shell>kubectl -n gitopsdemo-dev get secret &#60cluster_name-pguser-cluster_name> -o jsonpath="{.data.password}" | base64 -d; echo
</code></pre><p>You will notice that the password has been updated. Try this with the qa and prod applications as well. Changing the Postgres password has never been easier!<h3 id=postgres-upgrade><a href=#postgres-upgrade>Postgres Upgrade</a></h3><p>The PostgreSQL open source community typically releases a new major.minor version of Postgres every quarter. These releases will contain bug and CVE fixes. Crunchy Data updates their container images accordingly. It is considered a best practice to take these updates into your Postgres clusters running on Kubernetes as quickly as possible to apply CVE remediations and bug fixes.<p>Let’s check the current Postgres version in the dev cluster:<pre><code class=language-shell>$ kubectl exec -n gitopsdemo-dev hippo-pgdb-frxt-0 -c database -it -- bash
bash-4.4$ psql
psql (14.5)
Type "help" for help.

postgres=# select version();
                                                 version

---------------------------------------------------------------------------------------------------------
 PostgreSQL 14.5 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-10), 64-bit
(1 row)
</code></pre><p>Using Argo CD and GitOps we will declaratively state our intent by updating the images in our deployment artifacts in git. Let’s take a closer look.<p>In the <code>postgres.yaml</code> file in our base/postgres directory we have two images:<ul><li>Postgres: image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.5-1<li>pgBackrest: image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-2.40-1</ul><p>Update that file with the latest images for the same major version:<ul><li>Postgres: image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-14.6-0<li>pgBackrest: image: registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:ubi8-2.41-0</ul><p>Once you check these changes into your GitHub repo click the ‘Refresh Apps’ button at the top of the applications screen for the deploy project. You will see that the applications in the deploy project are now listed as ‘OutOfSynch’.<p>Select the deploy-dev application from the deploy project. Click on ‘Synch’ and then click on ‘Synchronize’ in the right panel.<p>When you synchronize the Postgres Operator will bring down the repo host and a Postgres replica pod. Those pods will immediately be restarted with the new images. Each Postgres replica pod will be upgraded one at a time until just the primary pod is left. The primary pod will then automatically failover to a recently upgraded replica pod and will then restart with the new image coming back up as a replica. If you only have one Postgres pod, as we do in the dev cluster, that pod immediately gets upgraded.<p>You will notice that the deploy-dev application is now marked asd ‘Synched’.<p>Now check the Postgres version:<pre><code class=language-shell>$ kubectl exec -n gitopsdemo-dev hippo-pgdb-frxt-0 -c database -it -- bash
bash-4.4$ psql
psql (14.6)
Type "help" for help.

postgres=# select version();
                                                 version

---------------------------------------------------------------------------------------------------------
 PostgreSQL 14.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15), 64-bit
(1 row)
</code></pre><p>Using this approach allows you to test your upgrade in the dev cluster before applying it to qa and prod. Once you feel confident that the Postgres cluster and its consumers are working as expected you can synchronize the deploy-qa application and then once qa testing is complete you can synchronize the deploy-prod application.<p>You changed the images in one file and were able to upgrade all three Postgres clusters thanks to Argo CD and GitOps.<h2 id=summary><a href=#summary>Summary</a></h2><p><a href=https://www.crunchydata.com/products/crunchy-postgresql-for-kubernetes>Crunchy Postgres for Kubernetes</a> is a powerful tool to rapidly deploy Postgres clusters in any Kubernetes environment. The declarative nature of CPK makes it a perfect candidate for GitOps. Administrative responsibilities don’t stop once the deployment is complete. Being able to perform Day 2 operations at scale is crucial to ensure the security, stability and performance of your Postgres clusters. Argo CD and GitOps can simplify the day to day operations needed by your enterprise deployments. This is just a small sample of what you can achieve with GitOps. If there are Kubernetes tasks that you run frequently or run against multiple clusters, GitOps may be worth a look. ]]></content:encoded>
<category><![CDATA[ Kubernetes ]]></category>
<author><![CDATA[ Bob.Pacheco@crunchydata.com (Bob Pacheco) ]]></author>
<dc:creator><![CDATA[ Bob Pacheco ]]></dc:creator>
<guid isPermalink="false">04f03015bf5e8a2420e1eabf0291d68c4b939253644e70b23633813be2e9ff73</guid>
<pubDate>Wed, 08 Feb 2023 10:00:00 EST</pubDate>
<dc:date>2023-02-08T15:00:00.000Z</dc:date>
<atom:updated>2023-02-08T15:00:00.000Z</atom:updated></item></channel></rss>