Accelerating Spatial Postgres: Varnish Cache for pg_tileserv using Kustomize

Recently I worked with one of my Crunchy Data PostgreSQL clients on implementing caching for pg_tileserv. pg_tileserv is a lightweight microservice to expose spatial data to the web and is a key service across many of our geospatial customer sites. pg_tileserv can generate a fair amount of database server load, depending on the complexity of the map data and number of end users, so putting a proxy cache such as Varnish in front of it is a best practice. Using Paul Ramsey's Production PostGIS Vector Tiles Caching as a starting point, I started to experiment with client standard toolset of Kustomize, OpenShift, and Podman. I found this was an easy and fast way to deploy Crunchy Data Kubernetes components and caching.

In this blog I would like to share with you a step-by-step guide on the process of deploying Varnish Caching for pg_tileserv using Kustomize in Redhat Openshift Container Platform and PGO, the Postgres Operator from Crunchy Data (PostgreSQL 13.x, PostGIS, pg_tileserv).

Prerequisites

• Access to Redhat Openshift Container Platform, with permission to deploy Kubernetes objects in the Namespace.
• Access to Openshift CLI 4.9 and knowledge of oc and Kubernetes commands.
• Access to deployed PostgreSQL Operator for Kubernetes (PGO) (PostgreSQL 13.x, PostGIS, pg_tileserv) environment able to view deployed components information.
• pg_tileserv pod/deployment, services are deployed, and in our environment pg_tileserv service is named tileserv.

Deployment Steps

Varnish Image

Find the right Varnish Cache image for your use case and upload it to your private client repository. Below are the steps we used in our client environment:

1. podman login registry.redhat.io
2. podman pull registry.redhat.io/rhel8/varnish-6:1-191.1655143360
3. podman tag registry.redhat.io/rhel8/varnish-6:1-191.1655143360 client registry/foldername/varnish-6:1-191.1655143360
4. podman push client registry/foldername/varnish-6:1-191.1655143360

Create the VCL file

Create the default.vcl file using a starting template from Varnish website. VCL stands for Varnish Configuration Language and is the domain-specific language used by Varnish to control request handling, routing, caching, and several other things.

Configure backend .host in default segment set to pg_tileserv service object, in our case it was tileserv.

• .host = tileserv
• .port = 7800

Sample default.vcl

vcl 4.0;

backend default { .host = "tileserv"; .port = "7800"; }

Create varnish_deployment.yaml

I will highlight few things that are important and provide our example below.

• Configure the image in the client private repository (client private_registry/ varnish-6:1-191.1655143360), use image to pull the secret if the private repository requires it.
• Mount tmpfs, mountPath /var/lib/varnish, Volumes emptyDir: {}.
• Volume mount the varnish configMap, to /etc/varnish/default.vcl.
• Configure containerPort to 8080.
• Configure Container resources, if this is not configured it will take default configuration configured at the cluster level, e.g. cpu request: 1M.
• Configure pod commands to execute varnishd and varnishncsa. varnishcsa command below adds varnish request info to the pod logs, this provides cache hit or miss information.

    command: ["/bin/sh"]
    args:
      - -c
      - |
      varnishd -a :8080 -a :6081 -f /etc/varnish/default.vcl -s malloc,512M;
      varnishncsa -F '%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" % Varnish:handling}x %{Varnish:hitmiss}x'

Sample Varnish deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: varnish
  labels:
  app: varnish
spec:
  replicas: 1
  selector:
  matchLabels:
    app: varnish
  template:
  metadata:
    labels:
    app: varnish
  spec:
    containers:
    - name: varnish
    image: privateclientregistry/crunchydata/varnish-6:1-191.1655143360
    imagePullPolicy: Always
    command: ["/bin/sh"]
    args:
      - -c
      - |
      varnishd -a :8080 -a :6081 -f /etc/varnish/default.vcl -s malloc,512M;
      varnishncsa -F '%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" % Varnish:handling}x %{Varnish:hitmiss}x'
    ports:
    - containerPort: 8080
    - containerPort: 6081
    resources:
      limits:
      cpu: 200m
      memory: 1024Mi
      requests:
      cpu: 100m
      memory: 512Mi
    volumeMounts:
    - mountPath: /etc/varnish/default.vcl
      name: varnish-config
      subPath: default.vcl
      defaultMode: 0777
    - mountPath: /var/lib/varnish
      name: tmpfs
      defaultMode: 0777
    volumes:
    - name: varnish-config
    configMap:
      name: varnish-config
      defaultMode: 0777
      items:
      - key: default.vcl
        path: default.vcl
    - name: tmpfs
    emptyDir: {}
    serviceAccount: pgo-default
    securityContext:
    runAsNonRoot: true
    fsGroupChangePolicy: "OnRootMismatch"
    terminationGracePeriodSeconds: 30

Create varnish_service.yaml for the Kubernetes varnish service with the ClusterIP

Below is our example:

---
kind: Service
apiVersion: v1
metadata:
  name: varnish-svc
  labels:
  name: varnish
spec:
  selector:
  app: varnish
  ports:
  - protocol: TCP
  port: 8080
  targetPort: 8080
  type: ClusterIP

Create varnish_route.yaml for a secure Varnish route

In our case we have created edge route with no certificates to enable https. Our recommendation is to follow your use case security guidelines and Openshift documentation. Below is an example:

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: varnish
  labels:
  name: varnish
spec:
  tls:
  insecureEdgeTerminationPolicy: Redirect
  termination: edge
  to:
  kind: Service
  name: varnish-svc
  weight: 100
  port:
  targetPort: 8080
  wildcardPolicy: None

Create kustomize.yaml

Here is an example manifest:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- varnish_deployment.yaml
- varnish_service.yaml
- varnish_route.yaml


configMapGenerator:
  - name: varnish-config
  files:
    - default.vcl

Deploy

The final step is to list files and deploy kustomize manifest using Openshift cli. Assumption here is that kustomize.yaml is in current directory.

$ ls
  # default.vcl
  # kustomization.yaml
  # varnish_deployment.yaml
  # varnish_route.yaml
  # varnish_service.yaml

oc apply -k .

Troubleshooting

Adjust Default VCL Cookies

After the initial deployment the Varnish pod log was showing most of the pages as a miss on the cache hit/miss. After analyzing, I found that set-cookie header was added to the request and response. Add default.vcl unset the cookies:

sub vcl_recv {
  if (req.url ~ "^[^?]*\.(pbf)(\?.*)?$") {
      unset req.http.Cookie;
    unset req.http.Authorization;
      # Only keep the following if VCL handling is complete
      return(hash);
  }
  }
  sub vcl_backend_response {
  if (bereq.url ~ "^[^?]*\.(pbf)(\?.*)?$") {
      unset beresp.http.Set-Cookie;
      set beresp.ttl = 1d;
  }
  }

Redeploy Varnish, assumption kustomize.yaml is in current directory.

# delete the deployment
oc delete -k .
# Deploy
oc apply -k .

Encountered Errors and Resolution

Closing Thoughts

Our clients have a complex mapping data. pg_tileserv was generating a fair amount of database server load and rendering could be slow at times. Implementing Varnish caching improved map rendering performance by 25% to 30% and equally reduced load on our database. I hope this blog was helpful, and we at Crunchy Data wish you happy learnings.