Kubernetes Bare-Metal: Resources Preparation

by Tohain Kuberneteson Posted on

Virtual Machines

The following Kubernetes infrastructure is using 3 physical servers which each runs virtual machines. The virtual machine servers can be any VM server of your choice such as Oracle VM. With Oracle VM server, there is one additional server needed to run Oracle VM Manager, so total of 4 physical servers need to be provided.

As best practice for data loss prevention, you also need an additional server with sufficient storage used as scheduled data backup destination.

Required virtual machines to run Kubernetes cluster is summarized in the following table.

VM Usage Location Specs IP Hostname
NFS 1 NFS storage first node Server 1 4 core/2 GB ram/1000 GB storage 10.0.0.2 nfs-one
NFS 2 NFS storage second node Server 2 4 core/2 GB ram/1000 GB storage 10.0.0.3 nfs-two
NFS 3 NFS storage third node Server 3 4 core/2 GB ram/1000 GB storage 10.0.0.4 nfs-three
ETCD 1 ETCD cluster first node Server 1 4 core/2 GB ram/20 GB storage 10.0.0.5 etcd-one
ETCD 2 ETCD cluster second node Server 2 4 core/2 GB ram/20 GB storage 10.0.0.6 etcd-two
ETCD 3 ETCD cluster third node Server 3 4 core/2 GB ram/20 GB storage 10.0.0.7 etcd-three
CP 1 Kubernetes Control Plane first node Server 1 4 core/4 GB ram/50 GB storage 10.0.0.11 k8s-cp-one
CP 2 Kubernetes Control Plane second node Server 2 4 core/4 GB ram/50 GB storage 10.0.0.12 k8s-cp-two
CP 3 Kubernetes Control Plane third node Server 3 4 core/4 GB ram/50 GB storage 10.0.0.13 k8s-cp-three
Worker 1 Kubernetes worker first node Server 1 16 core/32 GB ram/50 GB storage 10.0.0.14 k8s-node-one
Worker 2 Kubernetes worker second node Server 2 16 core/32 GB ram/50 GB storage 10.0.0.15 k8s-node-two
Worker 3 Kubernetes worker third node Server 2 16 core/32 GB ram/50 GB storage 10.0.0.16 k8s-node-three
Worker 4 Kubernetes worker fourth node Server 3 16 core/32 GB ram/50 GB storage 10.0.0.17 k8s-node-four
Worker 5 Kubernetes worker fifth node Server 3 16 core/32 GB ram/50 GB storage 10.0.0.18 k8s-node-five
Staging Apps staging repository Server 1 4 core/2 GB ram/50 GB storage 10.0.0.19 staging
Proxy Reverse proxy Server 1 4 core/2 GB ram/10 GB storage 10.0.0.20 proxy

Install an operating system on each virtual machines as you see fit, for consideration there are Debian or Ubuntu.

Certificate Authority

A TLS certificate is needed for ETCD cluster and Kubernetes Control Plane to operate. An easy way to generate certificate is using Easy RSA.

Acquire Easy RSA

From terminal execute the following commands. If you need to upgrade Easy RSA to latest version, also execute this commands by replacing the version.

cd ~
mkdir -p ca client
wget https://github.com/OpenVPN/easy-rsa/releases/download/v3.1.7/EasyRSA-3.1.7.tgz
tar -xvzf EasyRSA-3.1.7.tgz
cp -vR EasyRSA-3.1.7/* ca/
cp -vR EasyRSA-3.1.7/* client/
rm -rf EasyRSA-3.1.7

Setup CA

To initialize Public Key Infrastructure (PKI) for certificate authority, execute:

cd ~/ca
./easyrsa init-pki
Notice
------
'init-pki' complete; you may now create a CA or requests.

Your newly created PKI dir is:
* /home/user/ca/pki

Using Easy-RSA configuration:
* undefined

Next, build CA by issuing:

./easyrsa build-ca
No Easy-RSA 'vars' configuration file exists!

Using SSL:
* openssl OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

Enter New CA Key Passphrase:

Confirm New CA Key Passphrase:
...+.+..+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*................+..+......+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
....+.........+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+..+.........+.+...+...+...+.........+..+....+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+....+...+......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Common Name (eg: your user, host, or server name) [Easy-RSA CA]:My CA

Notice
------
CA creation complete. Your new CA certificate is at:
* /home/user/ca/pki/ca.crt

Issuing Certificate

First, initialize client PKI if its not already done, skip otherwise.

cd ~/client
./easyrsa init-pki
Notice
------
'init-pki' complete; you may now create a CA or requests.

Your newly created PKI dir is:
* /home/user/client/pki

Using Easy-RSA configuration:
* undefined

To issue a certificate, provides node hostname with ip address (etcd-one assumed) then follow this steps:

  • Generate certificate signing request

    cd ~/client
./easyrsa gen-req etcd-one nopass
    No Easy-RSA 'vars' configuration file exists!

Using SSL:
* openssl OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
.+....+......+..+...+.......+...+..+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+......+.+.....+.........+...+...+.......+......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+..........+..+.+..+................+.....+.......+..+..........+...........+.+.........+..+..........+..+...........................+.+..............+.+...+..+.......+......+..+....+...+........+...+....+...+..+.......+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
..+.......+..+......+.........+......+.......+.....+....+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.....+......+...............+..........+............+...+..+.+...........+....+..+...+...............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..+...+.+..+............+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Common Name (eg: your user, host, or server name) [etcd-one]:

Notice
------
Private-Key and Public-Certificate-Request files created.
Your files are:
* req: /home/user/client/pki/reqs/etcd-one.req
* key: /home/user/client/pki/private/etcd-one.key

  • Import certificate signing request

    cd ~/ca
./easyrsa import-req ../client/pki/reqs/etcd-one.req etcd-one
    No Easy-RSA 'vars' configuration file exists!

Using SSL:
* openssl OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

Notice
------
Request successfully imported with short-name: etcd-one
This request is now ready to be signed.

  • Sign certificate, if it's a peer replace server with serverClient

    ./easyrsa --subject-alt-name='DNS:etcd-one,DNS:10.0.0.5,IP:10.0.0.5' sign-req server etcd-one
    No Easy-RSA 'vars' configuration file exists!

Using SSL:
* openssl OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
You are about to sign the following certificate:
Please check over the details shown below for accuracy. Note that this request
has not been cryptographically verified. Please be sure it came from a trusted
source or that you have verified the request checksum with the sender.
Request subject, to be signed as a server certificate
for '825' days:

subject=
  commonName                = etcd-one

X509v3 Subject Alternative Name:
  DNS:etcd-one,DNS:10.0.0.5,IP:10.0.0.5

Type the word 'yes' to continue, or any other input to abort.
Confirm request details: yes

Using configuration from /home/user/ca/pki/openssl-easyrsa.cnf
Enter pass phrase for /home/user/ca/pki/private/ca.key:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
commonName            :ASN.1 12:'etcd-one'
Certificate is to be certified until Apr  1 10:44:00 2026 GMT (825 days)

Write out database with 1 new entries
Data Base Updated

Notice
------
Certificate created at:
* /home/user/ca/pki/issued/etcd-one.crt

  • Certiticate now available at the following locations

    Location Description
    ca/pki/ca.crt CA public key
    ca/pki/issued/etc-one.crt Certificate public key
    client/pki/private/etcd-one.key Certificate private key

    To view certificate details:

    openssl x509 -noout -text -in pki/issued/etcd-one.crt
    Certificate:
  Data:
      Version: 3 (0x2)
      Serial Number:
          bd:9c:99:96:29:80:cf:1f:39:29:2e:20:04:2b:77:40
      Signature Algorithm: sha256WithRSAEncryption
      Issuer: CN = My CA
      Validity
          Not Before: Dec 28 10:44:00 2023 GMT
          Not After : Apr  1 10:44:00 2026 GMT
      Subject: CN = etcd-one
      Subject Public Key Info:
          Public Key Algorithm: rsaEncryption
              Public-Key: (2048 bit)
              Modulus:
                  00:b4:ac:43:e6:e0:62:5c:4c:1e:2b:87:93:34:34:
                  d9:cf:d4:1a:7f:b9:53:15:0f:cf:00:9d:04:d5:3e:
                  f6:34:17:65:d5:0a:3a:7f:02:73:c2:5d:e8:c0:1b:
                  18:cc:01:3d:91:80:e8:10:24:d2:04:f8:9e:c9:c2:
                  73:64:73:9e:02:b6:44:02:fb:71:88:03:fc:6c:9c:
                  d6:e3:57:77:df:5c:d6:83:f0:64:2b:6b:5e:01:24:
                  74:5e:e4:84:26:97:17:99:c3:8c:ad:5a:8f:e1:0d:
                  60:50:5d:6f:b1:4b:48:43:76:12:be:ed:f6:1e:f3:
                  26:d2:9a:a6:c8:01:fe:29:60:c7:e8:c4:c1:34:6e:
                  4c:54:8c:5e:35:fe:aa:cf:92:55:18:ef:7e:de:58:
                  2c:2e:a2:21:b9:f5:36:a4:89:54:92:28:c7:8c:c7:
                  e3:0d:78:a4:98:64:eb:6a:58:40:13:1d:dd:8e:c6:
                  cb:01:0c:40:c9:c5:ca:a9:84:9c:d3:40:d1:df:d8:
                  9b:66:29:26:43:60:58:b2:db:ea:bb:0a:70:d5:77:
                  9f:df:56:62:ad:e1:b1:94:c1:f6:75:54:60:66:3b:
                  3e:cf:af:11:7a:94:b5:bd:ff:f6:39:ad:09:03:ab:
                  45:c4:c2:02:5c:31:84:cd:cc:13:9a:d6:a0:c0:c7:
                  b0:a9
              Exponent: 65537 (0x10001)
      X509v3 extensions:
          X509v3 Basic Constraints:
              CA:FALSE
          X509v3 Subject Key Identifier:
              E0:15:71:5E:54:EA:78:BA:FE:F5:99:34:04:31:D7:0D:38:64:24:5D
          X509v3 Authority Key Identifier:
              keyid:9D:64:E6:BB:A6:A5:AA:B2:32:DA:27:E9:6E:37:70:E7:23:5B:8D:DC
              DirName:/CN=My CA
              serial:1D:33:46:60:6D:A9:93:6D:29:66:02:F7:BD:1F:4C:02:8A:F9:2B:70
          X509v3 Extended Key Usage:
              TLS Web Server Authentication
          X509v3 Key Usage:
              Digital Signature, Key Encipherment
          X509v3 Subject Alternative Name:
              DNS:etcd-one, DNS:10.0.0.5, IP Address:10.0.0.5
  Signature Algorithm: sha256WithRSAEncryption
  Signature Value:
      34:72:fa:01:3e:76:54:4b:1b:04:97:04:01:c2:90:03:97:ba:
      d3:27:d9:96:f1:b9:9b:81:56:62:ab:4f:5e:2d:dd:db:ca:80:
      91:ba:9b:0f:20:07:c9:68:48:3e:62:d0:28:90:a9:4f:c9:15:
      77:50:ff:c8:2f:85:32:c8:62:e3:2d:7d:df:6d:0b:58:d8:71:
      2f:20:35:4e:4f:36:34:fe:33:e7:62:59:2a:1b:2f:8d:a4:25:
      df:fc:a9:f4:bf:c4:a8:c7:69:cb:22:ec:1c:01:d3:bc:06:9f:
      dd:1f:bf:de:51:92:44:9f:62:97:ed:d7:fe:77:66:f3:b9:07:
      ac:46:15:00:2a:23:8d:9c:44:db:ff:0d:36:13:1b:eb:0a:80:
      52:b3:0b:eb:2e:a8:50:79:7f:04:65:d6:99:fa:15:b8:dc:f8:
      15:27:05:91:fe:17:f5:a2:c9:8c:dc:13:d4:28:5e:44:eb:2b:
      c2:13:e0:03:ea:89:07:b3:d7:66:ec:66:d3:49:1c:93:01:18:
      86:98:3a:1e:4c:95:38:96:32:c1:8d:42:cc:86:ab:81:0e:56:
      6b:cd:07:15:ad:23:6f:48:ad:e6:b0:31:4d:9e:aa:63:d2:c5:
      11:72:c4:fa:d6:d3:69:91:a4:4a:53:97:4d:48:25:a5:71:bb:
      46:a9:10:19

The Certificates

The certificates need to be generated for ETCD cluster and Kubernetes is shown in the following table.

Name Type
etcd-one server
etcd-one-peer serverClient
etcd-two server
etcd-two-peer serverClient
etcd-three server
etcd-three-peer serverClient
k8s-cp-endpoint serverClient

NFS Storage

To setup each NFS server, follow this steps:

  • Ensure storage is already mounted, in this example /mnt/data is used

  • Install nfs-kernel-server

    sudo apt-get install nfs-kernel-server

  • Export shared storage

    sudo vi /etc/exports
    # /etc/exports: the access control list for filesystems which may be exported
#               to NFS client.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#
/mnt/data 10.0.0.0/24(rw,async,no_root_squash)

  • Restart nfs-server service and query for it's status

    sudo systemctl restart nfs-server.service
    sudo systemctl status nfs-server.service
    ● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; preset: enabled)
  Drop-In: /run/systemd/generator/nfs-server.service.d
           └─order-with-mounts.conf
   Active: active (exited) since Mon 2023-12-04 02:01:45 WIB; 3 weeks 3 days ago
  Process: 602 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
  Process: 603 ExecStart=/usr/sbin/rpc.nfsd (code=exited, status=0/SUCCESS)
 Main PID: 603 (code=exited, status=0/SUCCESS)
      CPU: 6ms

Dec 04 02:01:44 nfs-one exportfs[602]: exportfs: /etc/exports [1]: Neither 'subtree_check' or 'no_subtree_check' specified for export "10.0.0.0/24:/mnt/data".
Dec 04 02:01:44 nfs-one exportfs[602]:   Assuming default behaviour ('no_subtree_check').
Dec 04 02:01:44 nfs-one exportfs[602]:   NOTE: this default has changed since nfs-utils version 1.0.x
Dec 04 02:01:45 nfs-one systemd[1]: Finished nfs-server.service - NFS server and services.

What's Next

→ ETCD cluster setup
← Getting started

Leave a Reply