WordPress on Kubernetes

The Definitive Guide to WordPress on k8s

The MariaDB Operator

In this section we’ll introduce the concept of operators in Kubernetes, and use the MariaDB Operator to create and manage a scalable MariaDB database service in our Kubernetes cluster.

Note: all code samples from this section are available on GitHub.

Operators

Kubernetes operators are designed to automate manual work that’s needed to run a specific application in the cluster. These are typically used with stateful applications, such as databases, where every replica has its own state and identity.

Operators can take care of a wide range of tasks related to the application, such as creating and restoring backups, performing upgrades, updating configurations, handling failure and more.

Many operators in Kubernetes are built around Custom Resources (or CRDs) which allow us to extend the Kubernetes API with our own objects alongside the built-in resources, such as Pods.

On their own, custom resources aren’t very useful as they’re simply a place to store some structured data, however with a controller, CRDs can become very handy as you will see with the MariaDB example.

The MariaDB Operator

The MariaDB Operator for Kubernetes is a set of custom resource definitions and controllers, which allows users to create and manage MariaDB clusters.

Given these CRDs, instead of using StatefulSets and Services in Kubernetes, we’ll be defining objects using various new abstractions, such as MariaDBs, Databases, Users and Backups. The operator controllers will then take care of mapping these objects into any relevant Pods, Services and other Kubernetes built-in objects.

The MariaDB operator has a ton of useful features and we’ll be exploring just a few of those in this section.

Installing the Operator

In our previous sections we’ve relied on mariadb.statefulset.yml for our StatefulSet declaration, mariadb.service.yml for the Kubernetes Service and mariadb.secrets.yml to store our credentials.

Before installing the MariaDB operator, let’s destroy our existing MariaDB objects to avoid any confusion:

$ kubectl delete -f mariadb.secrets.yml \
  -f mariadb.service.yml \
  -f mariadb.statefulset.yml
secret "mariadb-secrets" deleted
service "mariadb" deleted
statefulset.apps "mariadb" deleted

Now let’s use Helm to install the MariaDB Operator:

$ helm repo add mariadb-operator https://helm.mariadb.com/mariadb-operator
$ helm install mariadb-operator mariadb-operator/mariadb-operator

You’ll see a few new running pods prefixed with “mariadb-operator”, these are the controllers that will watch for changes in our custom resources and other things.

Creating a MariaDB

Now that the operator is up and running, let’s re-create our MariaDB secrets first. We removed the database name and user name, as those will derive from our new resources later.

apiVersion: v1
kind: Secret
metadata:
  name: mariadb-secrets
  labels:
    k8s.mariadb.com/watch:
stringData:
  MARIADB_PASSWORD: secret
  MARIADB_ROOT_PASSWORD: verysecret

Note the new k8s.mariadb.com/watch label attached to this secret. This will ask the operator to monitor this for any changes, and perform updates when necessary. Note that updating the root password will require some hoops to jump through, but updating the WordPress user password will work just fine!

Let’s add this mariadb.secrets.yml to our cluster:

$ kubectl apply -f mariadb.secrets.yml
secret/mariadb-secrets created

Next, we’ll need a MariaDB resource, let’s call it mariadb.yml:

apiVersion: k8s.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: wordpress-mariadb
spec:
  rootPasswordSecretKeyRef:
    name: mariadb-secrets
    key: MARIADB_ROOT_PASSWORD
  storage:
    size: 1Gi
    storageClassName: openebs-hostpath

This is a custom resource with the kind MariaDB, which has a spec that supports a wide range of options and features. We used only a couple here, to specify the location of the root password, as well as a storage option to claim a local HostPath volume.

Note that in previous sections we used an OpenEBS/Mayastor replicated volume for some redundancy with MariaDB, but given that we’ll eventually be running multiple replicas this time around, it’s totally fine to downgrade this storage to a single OpenEBS non-replicated volume or even a local HostPath volume, for better performance.

Let’s apply this manifest and ensure our MariaDB resource and its related pods and service are up and running:

$ kubectl apply -f mariadb.yml                  
mariadb.k8s.mariadb.com/wordpress-mariadb created

$ kubectl get mariadbs
NAME                READY   STATUS    PRIMARY POD           AGE
wordpress-mariadb   True    Running   wordpress-mariadb-0   42s

$ kubectl get pods                    
NAME                                                READY   STATUS    RESTARTS   AGE
mariadb-operator-769bb76896-5z5md                   1/1     Running   0          78m
mariadb-operator-cert-controller-5d849657f4-lccxs   1/1     Running   0          78m
mariadb-operator-webhook-5d455b84d4-wk87v           1/1     Running   0          78m
minio-0                                             1/1     Running   0          82m
wordpress-756d97c644-pgk5k                          2/2     Running   0          82m
wordpress-mariadb-0                                 1/1     Running   0          49s

Our new pod is wordpress-mariadb-0 and you should also see a persistent volume claim for the pod, as well as a couple of ClusterIP services for this MariaDB deployment.

If the pod fails to start you should be able to obtain some information from the mariadb-operator pod logs.

Before we can use this with our WordPress installation, we’ll need to create a database, a user and a grant.

Databases, Users and Grants

As briefly mentioned earlier, the MariaDB operator includes custom resource definitions for databases, users and grants. This means that we can use YAML manifests to define and manage these resources.

Let’s squeeze them all into a single YAML file called mariadb.data.yml and separate the three resources with the --- block:

apiVersion: k8s.mariadb.com/v1alpha1
kind: Database
metadata:
  name: wordpress
spec:
  mariaDbRef:
    name: wordpress-mariadb

Our first resource is the Database named wordpress, linked to our MariaDB service using the mariaDbRef.name attribute.

---
apiVersion: k8s.mariadb.com/v1alpha1
kind: User
metadata:
  name: wordpress
spec:
  mariaDbRef:
    name: wordpress-mariadb
  passwordSecretKeyRef:
    name: mariadb-secrets
    key: MARIADB_PASSWORD
  maxUserConnections: 0

The next resource is a User also named wordpress, linked to our MariaDB service using the mariaDbRef.name attribute. We also set the maxUserConnections to unlimited here (only 10 by default) since we don’t really know how many connections our WordPress pods will need.

---
apiVersion: k8s.mariadb.com/v1alpha1
kind: Grant
metadata:
  name: wordpress
spec:
  mariaDbRef:
    name: wordpress-mariadb
  privileges:
  - ALL PRIVILEGES
  database: wordpress
  username: wordpress

Finally our Grant object linked to the same MariaDB service with ALL PRIVILEGES on the wordpress database granted to the wordpress user.

If you’ve created a user in MySQL or MariaDB before, all of this should not be a surprise. There is a short-hand to create all three as part of the MariaDB resource definition, that’s okay for testing purposes, but you’ll be stuck with the default connection limit of 10 which might be tricky to change later.

Let’s add these resources to our Kubernetes cluster:

$ kubectl apply -f mariadb.data.yml
database.k8s.mariadb.com/wordpress unchanged
user.k8s.mariadb.com/wordpress unchanged
grant.k8s.mariadb.com/wordpress created

We can use kubectl exec to make sure we can connect to our MariaDB database using these credentials:

$ kubectl exec -it wordpress-mariadb-0 -- \
  mysql -uwordpress -psecret wordpress -e 'show databases;'
+--------------------+
| Database           |
+--------------------+
| information_schema |
| wordpress          |
+--------------------+

MariaDB Replication

Setting up replication with the MariaDB operator in Kubernetes is quite straightforward. We need a replication.enabled flag and a minimum of two replicas.

With out traditional primary/replica configuration, we’ll also need separate endpoints to reach the primary server and the replica servers.

Our new mariadb.yml file will now look like this:

apiVersion: k8s.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: wordpress-mariadb
spec:
  rootPasswordSecretKeyRef:
    name: mariadb-secrets
    key: MARIADB_ROOT_PASSWORD

  storage:
    size: 1Gi
    storageClassName: openebs-hostpath

  replicas: 3
  replication:
    enabled: true

  primaryService:
    type: ClusterIP

  secondaryService:
    type: ClusterIP

The replicas and replication blocks tell the operator how many MariaDB pods we’d like to run. Note that the specified number of replicas includes the primary service in a primary/replica configuration, so the minimum number of two replicas will include one primary pod, and one replica. In the case above we’ll be running one primary and two replicas.

The primaryService and secondaryService attributes tell the operator what type of Services to create for our pods in the Kubernetes cluster. Let’s apply these changes to our Kubernetes cluster and observe the pods:

$ kubectl apply -f mariadb.yml
mariadb.k8s.mariadb.com/wordpress-mariadb configured

$ kubectl get pods
NAME                                                READY   STATUS    RESTARTS   AGE
mariadb-operator-769bb76896-5z5md                   1/1     Running   0          5h55m
mariadb-operator-cert-controller-5d849657f4-lccxs   1/1     Running   0          5h55m
mariadb-operator-webhook-5d455b84d4-wk87v           1/1     Running   0          5h55m
minio-0                                             1/1     Running   0          5h59m
wordpress-756d97c644-pgk5k                          2/2     Running   0          5h59m
wordpress-mariadb-0                                 1/1     Running   0          2m12s
wordpress-mariadb-1                                 1/1     Running   0          24s
wordpress-mariadb-2                                 1/1     Running   0          24s

We now have three MariaDB pods running, and quite a few services:

$ kubectl get svc
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
kubernetes                    ClusterIP   10.100.0.1       <none>        443/TCP          29d
mariadb-operator-webhook      ClusterIP   10.100.110.133   <none>        443/TCP          5h56m
minio                         ClusterIP   10.100.155.81    <none>        9000/TCP         12d
minio-console                 NodePort    10.100.192.4     <none>        9001:30008/TCP   12d
wordpress                     NodePort    10.100.152.147   <none>        80:30007/TCP     12d
wordpress-mariadb             ClusterIP   10.100.127.168   <none>        3306/TCP         3m1s
wordpress-mariadb-internal    ClusterIP   None             <none>        3306/TCP         3m1s
wordpress-mariadb-primary     ClusterIP   10.100.46.238    <none>        3306/TCP         3m8s
wordpress-mariadb-secondary   ClusterIP   10.100.77.21     <none>        3306/TCP         3m8s

The two services we’ll be using with WordPress are wordpress-mariadb-primary for writes and reads, and wordpress-mariadb-secondary for reads. Let’s try and query these services using the MySQL command line:

$ kubectl exec -it wordpress-mariadb-0 -- \
  mysql -uroot -pverysecret -hwordpress-mariadb-primary \
  -e 'show replica hosts;'
+-----------+-------------+------+-----------+
| Server_id | Host        | Port | Master_id |
+-----------+-------------+------+-----------+
|        12 | 10.10.2.235 | 3306 |        10 |
|        11 | 10.10.3.29  | 3306 |        10 |
+-----------+-------------+------+-----------+

We can also ensure that the secondary service connects us to a database that’s in read-only mode (i.e. a replica/slave):

$ kubectl exec -it wordpress-mariadb-0 -- \
  mysql -uroot -pverysecret -hwordpress-mariadb-secondary \
  -e "show variables like 'read_only';"
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only     | ON    |
+---------------+-------+

Finally, let’s make sure replication is actually working, by writing to a primary, then reading from a replica using our WordPress user credentials:

$ kubectl exec -it wordpress-mariadb-0 -- \
  mysql -uwordpress -psecret wordpress -hwordpress-mariadb-primary \
  -e 'create table foo (id integer);'

$ kubectl exec -it wordpress-mariadb-0 -- \
  mysql -uwordpress -psecret wordpress -hwordpress-mariadb-secondary \
  -e 'show tables;'
+---------------------+
| Tables_in_wordpress |
+---------------------+
| foo                 |
+---------------------+

At this point we can point our WordPress deployment to the wordpress-mariadb-primary database service via the wordpress.configmap.yml manifest, restart our deployment and run through the installation using just the primary database (we’ll explore working with replicas in WordPress in the next section).

Failover

As mentioned briefly earlier, one of the things the MariaDB operator helps us with is failover. Let’s delete one of the database replicas and see this in action:

$ kubectl delete pod wordpress-mariadb-2
pod "wordpress-mariadb-2" deleted

The underlying StatefulSet will create a new pod bound to the same existing persistent volume. The operator isn’t doing much work here since we’re essentially restarting the container from the same persistent volume. What if we deleted the persistent volume together with the pod?

$ kubectl delete pvc storage-wordpress-mariadb-1 \
  & kubectl delete pod wordpress-mariadb-1
persistentvolumeclaim "storage-wordpress-mariadb-1" deleted
pod "wordpress-mariadb-1" deleted

A few moments later you will see our operator created a new pod, with a brand new and empty persistent volume claim (and volume). If we inspect the replica status on the new pod, we’ll see that it’s all up-to-date and synced with the primary:

$ kubectl exec -it wordpress-mariadb-1 -- \
  mysql -uroot -pverysecret \
  -e 'show all replicas status\G'
*************************** 1. row ***************************
Connection_name: mariadb-operator
Slave_SQL_State: Slave has read all relay log; waiting for more updates
Slave_IO_State: Waiting for master to send event
# output omitted...

A more interesting experiment is to delete the primary node:

$ kubectl delete pvc storage-wordpress-mariadb-0 \
  & kubectl delete pod wordpress-mariadb-0
persistentvolumeclaim "storage-wordpress-mariadb-0" deleted
pod "wordpress-mariadb-0" deleted

At this stage, if you observe the MariaDB operator logs, you’ll see something along the lines of:

  • Configuring new primary
  • Connecting replicas to new primary
  • Primary switched
  • Configuring replica

The pod will get re-created like any other pod in a StatefulSet, however this time around it will no longer be a primary (by omitting the -h flag in mysql we’re connecting to the local database service, rather than going through a Kubernetes service):

$ kubectl exec -it wordpress-mariadb-0 -- \
  mysql -uroot -pverysecret -e 'show replica hosts'
# no output

$ kubectl exec -it wordpress-mariadb-0 -- \
  mysql -uroot -pverysecret -e "show variables like 'read_only';"       
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only     | ON    |
+---------------+-------+

$ kubectl exec -it wordpress-mariadb-1 -- \
mysql -uroot -pverysecret -e 'show replica hosts'
+-----------+-------------+------+-----------+
| Server_id | Host        | Port | Master_id |
+-----------+-------------+------+-----------+
|        10 | 10.10.3.254 | 3306 |        11 |
|        12 | 10.10.2.15  | 3306 |        11 |
+-----------+-------------+------+-----------+

In our case the wordpress-mariadb-1 pod was promoted to be the new primary and all replicas (including the new one) were configured to read from that instead.

Our Services have also been updated to make sure they’re continuing to point to the correct pods:

$ kubectl describe svc wordpress-mariadb-primary
Endpoints:         10.10.3.176:3306
# output omitted

In the case above, 10.10.3.176 is the IP address of our recently promoted wordpress-mariadb-1 pod that is now the primary. The secondary MariaDB Service has also been updated to point to the replica pods.

What’s next?

We encourage you to explore the examples directory of the MariaDB operator if you’d like to learn more about it. For our purposes here with WordPress, we have a fully working replicated MariaDB service in our Kubernetes cluster. We have primary and secondary service endpoints we can use to write and read to and from our database.

As mentioned earlier, in order to now make good use of this highly available database cluster, we’ll need to split read and write queries in WordPress. This can be done using a special plugin called HyperDB, which we’ll deep dive into next.