Kubernetes Bare-Metal: Operator and Provisioner Deployment

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.

Operator gives simplification to deploy common application such as database servers. In this example, there are MySQL Operator and MongoDB Community Operator. For shared storage, a NFS is used which is provisioned by NFS Subdir External Provisioner.

A configuration directory layout used in this page is shown as follows:

> kubernetes
  > config                                <- configurations
  > mongodb-operator                      <- MongoDB operator repository
  > mysql-operator                        <- MySQL operator repository
  > nfs-provisioner
    > nfs-subdir-external-provisioner     <- NFS subdir external provisioner repository
    > provisioner-one                     <- provisioner one
    > provisioner-two                     <- provisioner two
    > provisioner-three                   <- provisioner three
  > patches                               <- patches

NFS Provisioner

NFS provisioner is provided by NFS Subdir External Provisioner which allows us to use NFS server for persistent volume. NFS Subdir External Provisioner provides a boiler plate for a single NFS server, but in this setup there will be three NFS servers.

  • Clone NFS Subdir External Provisioner repository

    mkdir -p nfs-provisioner
    git clone https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.git nfs-provisioner/nfs-subdir-external-provisioner
  • Review provisioner configuration for provisioner-one, provisioner-two, and provisioner-three

    vi nfs-provisioner/provisioner-one/kustomization.yaml
    - deploy/class.yaml
    - deploy/deployment.yaml
    - patches.yaml
    - target:
      kind: StorageClass
      name: nfs-client
    patch: |-
      - op: replace
        path: /metadata/name
        value: nfs-data-1
      - op: replace
        path: /provisioner
        value: k8s-sigs.io/nfs-subdir-external-provisioner-one
    - target:
      kind: Deployment
      name: nfs-client-provisioner
    patch: |-
      - op: replace
        path: /metadata/name
        value: nfs-client-provisioner-one
      - op: replace
        path: /metadata/labels/app
        value: nfs-client-provisioner-one
      - op: replace
        path: /spec/selector/matchLabels/app
        value: nfs-client-provisioner-one
      - op: replace
        path: /spec/template/metadata/labels/app
        value: nfs-client-provisioner-one
    vi nfs-provisioner/provisioner-one/patches.yaml
    apiVersion: apps/v1
    kind: Deployment
      app: nfs-client-provisioner
    name: nfs-client-provisioner
          - name: nfs-client-provisioner
              - name: PROVISIONER_NAME
                value: k8s-sigs.io/nfs-subdir-external-provisioner-one
              - name: NFS_SERVER
              - name: NFS_PATH
                value: /mnt/data
          - name: nfs-client-root
              path: /mnt/data
  • Ensure NFS Subdir External Provisioner deployment configuration is copied accross all NFS server instances

    cat nfs-provisioner/copy-deployment.sh
    PROVISIONERS="provisioner-one provisioner-two provisioner-three"
      if [ -d "$PROVISIONER" ]; then
          rsync -av nfs-subdir-external-provisioner/deploy $PROVISIONER
  • Apply kustomization for our NFS provisioner

    kubectl apply -k nfs-provisioner
  • After the kustomization applied, there will be three storage classes

    • nfs-data-1 which belong to first NFS server (
    • nfs-data-2 which belong to second NFS server (
    • nfs-data-3 which belong to third NFS server (

MySQL Operator for Kubernetes

MySQL Operator for Kubernetes deployment is detailed from the official site. The operator will be deployed in the mysql-operator namespace.

  • Clone MySQL Operator repository

    git clone https://github.com/mysql/mysql-operator.git
  • Apply CRD

    kubectl apply -f mysql-operator/deploy/deploy-crds.yaml
    customresourcedefinition.apiextensions.k8s.io/innodbclusters.mysql.oracle.com created
    customresourcedefinition.apiextensions.k8s.io/mysqlbackups.mysql.oracle.com created
    customresourcedefinition.apiextensions.k8s.io/clusterkopfpeerings.zalando.org created
    customresourcedefinition.apiextensions.k8s.io/kopfpeerings.zalando.org created
  • Deploy operator

    kubectl apply -f mysql-operator/deploy/deploy-operator.yaml
    serviceaccount/mysql-sidecar-sa created
    clusterrole.rbac.authorization.k8s.io/mysql-operator created
    clusterrole.rbac.authorization.k8s.io/mysql-sidecar created
    clusterrolebinding.rbac.authorization.k8s.io/mysql-operator-rolebinding created
    clusterkopfpeering.zalando.org/mysql-operator created
    namespace/mysql-operator created
    serviceaccount/mysql-operator-sa created
    deployment.apps/mysql-operator created
  • Ensure the operator is running

    kubectl get pod -n mysql-operator -o wide
    NAME                             READY   STATUS    RESTARTS   AGE   IP             NODE            NOMINATED NODE   READINESS GATES
    mysql-operator-b6dd879db-kzphw   1/1     Running   0          49s   k8s-node-four              

MongoDB Community Kubernetes Operator

MongoDB Community Kubernetes Operator deployment is detailed from the official site. The following deployment will assume to use mongodb-operator namespace. See here for more information.

  • Clone MongoDB Community Kubernetes Operator repository

    git clone https://github.com/mongodb/mongodb-kubernetes-operator.git mongodb-operator
  • Apply patch to allow the operator to watch all namespaces and change the namespace accordingly

    vi patches/mongodb-kubernetes-operator.patch
    diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml
    index e488e07..a9e7921 100644
    --- a/config/manager/manager.yaml
    +++ b/config/manager/manager.yaml
    @@ -35,9 +35,7 @@ spec:
          - /usr/local/bin/entrypoint
          - name: WATCH_NAMESPACE
    -          valueFrom:
    -            fieldRef:
    -              fieldPath: metadata.namespace
    +          value: "*"
          - name: POD_NAME
    diff --git a/deploy/clusterwide/cluster_role_binding.yaml b/deploy/clusterwide/cluster_role_binding.yaml
    index 7617ec0..3f426e2 100644
    --- a/deploy/clusterwide/cluster_role_binding.yaml
    +++ b/deploy/clusterwide/cluster_role_binding.yaml
    @@ -4,7 +4,7 @@ metadata:
    name: mongodb-kubernetes-operator
    - kind: ServiceAccount
    -  # namespace: 
    +  namespace: mongodb-operator
    name: mongodb-kubernetes-operator
    kind: ClusterRole
    cd mongodb-operator && git apply ../patches/mongodb-kubernetes-operator.patch
  • Create namespace

    kubectl create namespace mongodb-operator
    namespace/mongodb-operator created
  • Install CRD

    kubectl apply -f mongodb-operator/config/crd/bases/mongodbcommunity.mongodb.com_mongodbcommunity.yaml
    customresourcedefinition.apiextensions.k8s.io/mongodbcommunity.mongodbcommunity.mongodb.com created
  • Deploy cluster role and binding

    kubectl apply -f mongodb-operator/deploy/clusterwide/
    clusterrole.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
    clusterrolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
    clusterrole.rbac.authorization.k8s.io/read-access-for-service-binding created
  • Install RBAC

    kubectl apply -k mongodb-operator/config/rbac/ -n mongodb-operator
    serviceaccount/mongodb-database created
    serviceaccount/mongodb-kubernetes-operator created
    role.rbac.authorization.k8s.io/mongodb-database created
    role.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
    rolebinding.rbac.authorization.k8s.io/mongodb-database created
    rolebinding.rbac.authorization.k8s.io/mongodb-kubernetes-operator created
  • Install operator

    kubectl apply -f mongodb-operator/config/manager/manager.yaml -n mongodb-operator
    deployment.apps/mongodb-kubernetes-operator created
  • Ensure the operator is running

    kubectl get pod -n mongodb-operator -o wide
    NAME                                           READY   STATUS    RESTARTS   AGE   IP            NODE           NOMINATED NODE   READINESS GATES
    mongodb-kubernetes-operator-7f4b4598dd-bkhbs   1/1     Running   0          70s   k8s-node-one              

What's Next

Apps deployment
Accessing cluster using Dashboard

Leave a Reply