Kubernetes Bare-Metal: Troubleshooting

by Tohain Kuberneteson Posted on

Recover MySQL Cluster After Unexpected Shutdown

When MySQL Server shutdown unexpectedly there will be possibility the cluster will be broken and MySQL Server becomes read only.

  • Get rid of MySQL read only status

    kubectl exec -it -n myapp -c mysql mysql-0 -- mysql -u root -p
    show variables like '%read_only%';
    +-----------------------+-------+
| Variable_name         | Value |
+-----------------------+-------+
| innodb_read_only      | OFF   |
| read_only             | ON    |
| super_read_only       | ON    |
| transaction_read_only | OFF   |
+-----------------------+-------+
    set global super_read_only=off;
set global read_only=off;

  • Reboot cluster

    kubectl exec -it -n myapp -c sidecar mysql-0 -- mysqlsh -u root -p
    dba.rebootClusterFromCompleteOutage()

MySQL Cluster mysql-privsecrets Secret Which Holds MySQL Cluster Admin Username and Password is Dynamically Created Each Time the Cluster Deployed

While mysql users and passwords are saved in mysql.user table, once the cluster is deleted and then re-deployed, password for mysqladmin is out of sync and lead the cluster to be unhealthy. The workaround is to sync those password manually.

  • Get the current password using dashboard or via shell

    kubectl get secret mysql-privsecrets -n myapp -o go-template="{{.data.clusterAdminPassword | base64decode}}" && echo ""

  • Login to the container

    kubectl exec -it -n myapp -c mysql mysql-0 -- mysql -u root -p
    set sql_log_bin=off;
alter user 'mysqladmin'@'%' identified by 'the-new-password-above';

What's Next

← Scheduled backup

Leave a Reply