Kubernetes Bare-Metal: Operator and Provisioner Deployment

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.

Operator gives simplification to deploy common application such as database servers. In this example, there are MySQL Operator and MongoDB Community Operator. For shared storage, a NFS is used which is provisioned by NFS Subdir External Provisioner.

A configuration directory layout used in this page is shown as follows:

> kubernetes
  > config                                <- configurations
  > mongodb-operator                      <- MongoDB operator repository
  > mysql-operator                        <- MySQL operator repository
  > nfs-provisioner
    > nfs-subdir-external-provisioner     <- NFS subdir external provisioner repository
    > provisioner-one                     <- provisioner one
    > provisioner-two                     <- provisioner two
    > provisioner-three                   <- provisioner three
  > patches                               <- patches

NFS Provisioner

NFS provisioner is provided by NFS Subdir External Provisioner which allows us to use NFS server for persistent volume. NFS Subdir External Provisioner provides a boiler plate for a single NFS server, but in this setup there will be three NFS servers.

Clone NFS Subdir External Provisioner repository

mkdir -p nfs-provisioner
git clone https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.git nfs-provisioner/nfs-subdir-external-provisioner

Review provisioner configuration for provisioner-one, provisioner-two, and provisioner-three

vi nfs-provisioner/provisioner-one/kustomization.yaml

resources:
- deploy/class.yaml
- deploy/deployment.yaml
patchesStrategicMerge:
- patches.yaml
patches:
- target:
  kind: StorageClass
  name: nfs-client
patch: |-
  - op: replace
    path: /metadata/name
    value: nfs-data-1
  - op: replace
    path: /provisioner
    value: k8s-sigs.io/nfs-subdir-external-provisioner-one
- target:
  kind: Deployment
  name: nfs-client-provisioner
patch: |-
  - op: replace
    path: /metadata/name
    value: nfs-client-provisioner-one
  - op: replace
    path: /metadata/labels/app
    value: nfs-client-provisioner-one
  - op: replace
    path: /spec/selector/matchLabels/app
    value: nfs-client-provisioner-one
  - op: replace
    path: /spec/template/metadata/labels/app
    value: nfs-client-provisioner-one

vi nfs-provisioner/provisioner-one/patches.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
labels:
  app: nfs-client-provisioner
name: nfs-client-provisioner
spec:
template:
  spec:
    containers:
      - name: nfs-client-provisioner
        env:
          - name: PROVISIONER_NAME
            value: k8s-sigs.io/nfs-subdir-external-provisioner-one
          - name: NFS_SERVER
            value: 10.0.0.2
          - name: NFS_PATH
            value: /mnt/data
    volumes:
      - name: nfs-client-root
        nfs:
          server: 10.0.0.2
          path: /mnt/data

Ensure NFS Subdir External Provisioner deployment configuration is copied accross all NFS server instances

cat nfs-provisioner/copy-deployment.sh

#!/bin/bash

DIR=pwd
PROVISIONERS="provisioner-one provisioner-two provisioner-three"
for PROVISIONER in $PROVISIONERS; do
  if [ -d "$PROVISIONER" ]; then
      rsync -av nfs-subdir-external-provisioner/deploy $PROVISIONER
  fi 
done

./nfs-provisioner/copy-deployment.sh

Apply kustomization for our NFS provisioner
```
kubectl apply -k nfs-provisioner
```
After the kustomization applied, there will be three storage classes
- nfs-data-1 which belong to first NFS server (10.0.0.2)
- nfs-data-2 which belong to second NFS server (10.0.0.3)
- nfs-data-3 which belong to third NFS server (10.0.0.4)

MySQL Operator for Kubernetes

MySQL Operator for Kubernetes deployment is detailed from the official site. The operator will be deployed in the mysql-operator namespace.

Clone MySQL Operator repository

git clone https://github.com/mysql/mysql-operator.git

Apply CRD

kubectl apply -f mysql-operator/deploy/deploy-crds.yaml

customresourcedefinition.apiextensions.k8s.io/innodbclusters.mysql.oracle.com created
customresourcedefinition.apiextensions.k8s.io/mysqlbackups.mysql.oracle.com created
customresourcedefinition.apiextensions.k8s.io/clusterkopfpeerings.zalando.org created
customresourcedefinition.apiextensions.k8s.io/kopfpeerings.zalando.org created

Deploy operator

kubectl apply -f mysql-operator/deploy/deploy-operator.yaml

serviceaccount/mysql-sidecar-sa created
clusterrole.rbac.authorization.k8s.io/mysql-operator created
clusterrole.rbac.authorization.k8s.io/mysql-sidecar created
clusterrolebinding.rbac.authorization.k8s.io/mysql-operator-rolebinding created
clusterkopfpeering.zalando.org/mysql-operator created
namespace/mysql-operator created
serviceaccount/mysql-operator-sa created
deployment.apps/mysql-operator created

Ensure the operator is running

kubectl get pod -n mysql-operator -o wide

NAME                             READY   STATUS    RESTARTS   AGE   IP             NODE            NOMINATED NODE   READINESS GATES
mysql-operator-b6dd879db-kzphw   1/1     Running   0          49s   10.244.8.194   k8s-node-four

MongoDB Community Kubernetes Operator

MongoDB Community Kubernetes Operator deployment is detailed from the official site. The following deployment will assume to use mongodb-operator namespace. See here for more information.

Clone MongoDB Community Kubernetes Operator repository

git clone https://github.com/mongodb/mongodb-kubernetes-operator.git mongodb-operator

Apply patch to allow the operator to watch all namespaces and change the namespace accordingly

vi patches/mongodb-kubernetes-operator.patch

diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml
index e488e07..a9e7921 100644
--- a/config/manager/manager.yaml
+++ b/config/manager/manager.yaml
@@ -35,9 +35,7 @@ spec:
      - /usr/local/bin/entrypoint
      env:
      - name: WATCH_NAMESPACE
-          valueFrom:
-            fieldRef:
-              fieldPath: metadata.namespace
+          value: "*"
      - name: POD_NAME
        valueFrom:
          fieldRef:
diff --git a/deploy/clusterwide/cluster_role_binding.yaml b/deploy/clusterwide/cluster_role_binding.yaml
index 7617ec0..3f426e2 100644
--- a/deploy/clusterwide/cluster_role_binding.yaml
+++ b/deploy/clusterwide/cluster_role_binding.yaml
@@ -4,7 +4,7 @@ metadata:
name: mongodb-kubernetes-operator
subjects:
- kind: ServiceAccount
-  # namespace: 
+  namespace: mongodb-operator
name: mongodb-kubernetes-operator
roleRef:
kind: ClusterRole

cd mongodb-operator && git apply ../patches/mongodb-kubernetes-operator.patch

Create namespace

kubectl create namespace mongodb-operator

namespace/mongodb-operator created

Install CRD

kubectl apply -f mongodb-operator/config/crd/bases/mongodbcommunity.mongodb.com_mongodbcommunity.yaml

customresourcedefinition.apiextensions.k8s.io/mongodbcommunity.mongodbcommunity.mongodb.com created

Deploy cluster role and binding

kubectl apply -f mongodb-operator/deploy/clusterwide/

clusterrole.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
clusterrolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
clusterrole.rbac.authorization.k8s.io/read-access-for-service-binding created

Install RBAC

kubectl apply -k mongodb-operator/config/rbac/ -n mongodb-operator

serviceaccount/mongodb-database created
serviceaccount/mongodb-kubernetes-operator created
role.rbac.authorization.k8s.io/mongodb-database created
role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
rolebinding.rbac.authorization.k8s.io/mongodb-database created
rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created

Install operator

kubectl apply -f mongodb-operator/config/manager/manager.yaml -n mongodb-operator

deployment.apps/mongodb-kubernetes-operator created

Ensure the operator is running

kubectl get pod -n mongodb-operator -o wide

NAME                                           READY   STATUS    RESTARTS   AGE   IP            NODE           NOMINATED NODE   READINESS GATES
mongodb-kubernetes-operator-7f4b4598dd-bkhbs   1/1     Running   0          70s   10.244.49.1   k8s-node-one

What's Next

→ Apps deployment
← Accessing cluster using Dashboard

Kubernetes Bare-Metal: Operator and Provisioner Deployment

NFS Provisioner

MySQL Operator for Kubernetes

MongoDB Community Kubernetes Operator

What's Next

Like this:

Leave a ReplyCancel reply

NFS Provisioner

MySQL Operator for Kubernetes

MongoDB Community Kubernetes Operator

What's Next

Share this:

Like this:

Leave a ReplyCancel reply