Netris Controller Maintenance and Backups

This page covers day-2 operation of an HA Netris Controller: taking nodes down for maintenance safely, and locating, verifying, and restoring MariaDB backups. To install a new controller, see Installing HA Netris Controller in Air-Gapped Environments. To upgrade an existing one, see Upgrading the HA Netris Controller.

Maintenance Procedures 

Proper maintenance procedures are critical for ensuring the continued stability and availability of your Netris Controller HA deployment. Improper shutdown or maintenance sequences can lead to database cluster inconsistencies, particularly with MariaDB, potentially resulting in service disruptions or data corruption.

Node Maintenance Best Practices 

Individual Node Maintenance (Recommended Approach)

The safest approach is to perform maintenance on one node at a time, keeping the cluster operational throughout the process:

Identify the primary MariaDB node before starting maintenance:
```
kubectl -nnetris-controller get maxscale netris-controller-ha-mariadb
```
Note the PRIMARY column output (e.g., netris-controller-ha-mariadb-ha-0)
Find which physical node is hosting the primary MariaDB:
```
kubectl -nnetris-controller get pod netris-controller-ha-mariadb-ha-0 -o wide
```
Note the NODE column (e.g., ctl-ha-node1)
Plan your maintenance order:
- Start with nodes NOT hosting the primary MariaDB
- Leave the node hosting the primary MariaDB for last
For each non-primary node:
1. Cordon the node to prevent new pods from being scheduled:
```
kubectl cordon <node-name>
```
2. Drain the node safely to relocate all running pods:
```
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
```
3. Verify pods have been relocated:
```
kubectl get pods -A -o wide | grep <node-name>
```
4. Perform maintenance on the node (updates, reboots, etc.)
5. Bring the node back online.
6. Verify the node is ready:
```
kubectl get nodes
```
7. Uncordon the node:
```
kubectl uncordon <node-name>
```
8. Verify cluster health before proceeding to the next node:
```
kubectl get pods -n netris-controller
kubectl -nnetris-controller get maxscale netris-controller-ha-mariadb
```
For the node hosting the primary MariaDB:
1. Double-check it’s still hosting the primary (as failover might have occurred):
```
kubectl -nnetris-controller get maxscale netris-controller-ha-mariadb
kubectl -nnetris-controller get pod <primary-pod-name> -o wide
```
2. Follow the same cordon, drain, maintenance, and uncordon steps as above

Full Cluster Maintenance (When All Nodes Need Simultaneous Maintenance)

If you need to shut down multiple nodes simultaneously:

Identify the primary MariaDB node:
```
kubectl -nnetris-controller get maxscale netris-controller-ha-mariadb
```
Note the PRIMARY column output (e.g., netris-controller-ha-mariadb-ha-0)

Find which physical nodes are hosting each MariaDB instance:

kubectl -nnetris-controller get pod -l app.kubernetes.io/name=mariadb -o wide

Safe node shutdown sequence:

Shutdown secondary/replica nodes first:

# For each non-primary node
kubectl cordon <non-primary-node>
kubectl drain <non-primary-node> --ignore-daemonsets --delete-emptydir-data
# Wait at least 1 minute before shutting down or proceeding to next node
sudo shutdown -h now  # Only on the drained node

Shutdown the primary node last:

kubectl cordon <primary-node>
kubectl drain <primary-node> --ignore-daemonsets --delete-emptydir-data
sudo shutdown -h now  # Only on the primary node

Safe node startup sequence:
1. Start the node that was hosting the primary MariaDB first
2. Wait until it’s fully online (check with kubectl get nodes)
3. Start the remaining nodes one by one, with at least 2 minutes between each
4. Uncordon each node after it’s online:
```
kubectl uncordon <node-name>
```

Verify cluster health:

kubectl get nodes
kubectl -n netris-controller get pods
kubectl -nnetris-controller get maxscale netris-controller-ha-mariadb

Rebalance pods across all nodes:

After all nodes are back online and uncordoned, restart all deployments to ensure even pod distribution:
```
# This will restart all deployments in netris-controller namespace
kubectl -nnetris-controller rollout restart deployment
```
Wait for all pods to restart and reach Running state:
```
kubectl -nnetris-controller get pods
```
Verify that pods are now distributed evenly across all nodes:
```
kubectl -nnetris-controller get pods -o wide
```

Verifying MariaDB Cluster Health 

After maintenance, verify the MariaDB cluster is healthy:

Check MaxScale status:
```
kubectl -nnetris-controller get maxscale netris-controller-ha-mariadb
```
The STATUS should show Running, and a PRIMARY should be identified

Verify all MariaDB pods are running:

kubectl -n netris-controller get pods -l app.kubernetes.io/name=mariadb

If issues are detected, check the operator logs:

kubectl -n netris-controller logs -l app.kubernetes.io/name=mariadb-operator

Maintenance Best Practices 

Always perform one-node-at-a-time maintenance when possible
Never power off nodes without properly cordoning and draining
Always shut down secondary/replica database nodes before the primary
Always start the primary node first when bringing the system back online
Verify cluster health after each node completes maintenance
Rebalance your workloads by restarting deployments after all maintenance is complete
Schedule maintenance during low-usage periods
Create a backup before maintenance
Document all maintenance activities in a maintenance log

MariaDB automatic backups: locate, verify, and restore 

The Netris Controller automatically creates MariaDB backups every 12 hours.

These backups are stored locally on each controller node.

Warning

Because backups are stored on local disks, copy them to an external and secure location such as object storage, NFS, or a backup server for disaster recovery.

Locate the backup directory on each controller node 

Each controller node stores a MariaDB backup locally.

Run the following command locally on each controller node. It detects and exports the backup directory path for the current node only:

export BACKUP_PATH=$(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{"|"}{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}{"|"}{.spec.local.path}{"\n"}{end}' \
| grep netris-controller-ha-mariadb-backup \
| grep "|$(hostname | tr '[:upper:]' '[:lower:]')|" \
| cut -d'|' -f3)

Example:

ubuntu@ctl-ha-node1:~$ sudo ls -al $BACKUP_PATH
total 296
drwxrwsrwx 2 root  999   4096 Feb  3 07:40 .
drwx------ 9 root root   4096 Feb  3 07:40 ..
-rw-r--r-- 1 lxd   999     39 Feb  3 07:40 0-backup-target.txt
-rw-r--r-- 1 lxd   999 289736 Feb  3 07:40 backup.2026-02-03T07:40:03Z.sql

Copy the backup file to your home directory 

Choose the required backup file and copy it to your home directory:

sudo cp $BACKUP_PATH/backup.2026-02-03T07:40:03Z.sql ~/backup.sql

Verify the backup file integrity 

Before restoring a backup, verify that the dump completed successfully.

The last line of the dump file must contain:

-- Dump completed on <date>

Check the last line by running:

tail ~/backup.sql

Example output:

-- Dump completed on 2026-02-03  7:40:03

Caution

If this line is missing, do not restore from this backup.

Verify MariaDB cluster readiness before restore 

Before restoring, confirm that the MariaDB cluster is in the READY state:

kubectl -n netris-controller get maxscale netris-controller-ha-mariadb

Expected output:

NAME                           READY   STATUS    PRIMARY                             AGE
netris-controller-ha-mariadb   True    Running   netris-controller-ha-mariadb-ha-0   66m

Proceed only if READY is True.

Copy the backup file into a MariaDB pod 

Copy the backup file into one of the MariaDB pods:

kubectl -n netris-controller cp ~/backup.sql \
  netris-controller-ha-mariadb-ha-0:/tmp/backup.sql

No output is expected.

Restore the backup 

Run the restore command inside the MariaDB pod:

kubectl -n netris-controller exec -it netris-controller-ha-mariadb-ha-0 -- \
  bash -c 'mysql -h netris-controller-ha-mariadb -u netris -pchangeme netris < /tmp/backup.sql'

No output is expected if the restore completes successfully.

Take a manual backup 

To create a MariaDB backup manually at any time, run:

kubectl -n netris-controller exec -it netris-controller-ha-mariadb-ha-0 -- \
  bash -c 'mysqldump -h netris-controller-ha-mariadb -u netris -pchangeme netris' \
  > db-snapshot-$(date +%Y-%m-%d-%H-%M-%S).sql

Summary 

MariaDB backups are created automatically every 12 hours.
Backups are stored locally on each controller node.
Copy backups to external storage for disaster recovery.
Verify backup integrity before restoring.
Restore only when the MariaDB cluster is in the READY state.

For serious database issues, contact Netris support with:

Output of kubectl -nnetris-controller get maxscale netris-controller-ha-mariadb -o yaml
Logs from MariaDB pods and operator

By following these maintenance procedures, you can significantly reduce the risk of database inconsistencies and service disruptions during and after maintenance operations.