I'm trying to backup a MySQL database running inside a k8s cluster (k3s). This cluster is running locally on a Raspberry Pi. I've built a custom image based on Alpine Linux that contains a script to create the backup using mysqldump:
kubectl exec <pod_name> -n <namespace_name> -- /usr/bin/mysqldump -u <db_user> --password=<db_password> --verbose <db_name> > <file_name>
When I run the mysqldump command from inside the database pod, it completes successfully after 10-15 seconds. But when this command gets executed from inside the Alpine pod it somehow takes a lot longer (2m40s). At that point it stops/aborts the kubectl exec command (due to a timeout?) and the script uploads a corrupt sql file because the mysqldump command wasn't finished with the backup.
Expected verbose output:
-- Connecting to localhost...
-- Retrieving table structure for table _prisma_migrations...
-- Sending SELECT query...
-- Retrieving rows...
-- Retrieving table structure for table database...
-- Sending SELECT query...
-- Retrieving rows...
-- Retrieving table structure for table recycleBin...
-- Sending SELECT query...
-- Retrieving rows...
-- Retrieving table structure for table user...
-- Sending SELECT query...
-- Retrieving rows...
-- Disconnecting from localhost...
Received verbose output:
-- Connecting to localhost...
-- Retrieving table structure for table _prisma_migrations...
-- Sending SELECT query...
-- Retrieving rows...
-- Retrieving table structure for table database...
-- Sending SELECT query...
-- Retrieving rows...
-- Retrieving table structure for table recycleBin...
-- Sending SELECT query...
-- Retrieving rows...
I have 2 questions:
- Why does the mysqldump command take so much longer in the Alpine pod compared to the database pod?
- Why isn't the kubectl exec command waiting until the mysqldump command has finished taking the backup? Why does it suddenly decide it's time to disconnect and move on?
CodePudding user response：
It is possible that you're getting disconnected because kubectl thinks the connection is dead due to no data from the other side.
Instead of:
kubectl exec <pod_name> -n <namespace_name> -- /usr/bin/mysqldump -u <db_user> --password=<db_password> --verbose <db_name> > <file_name>
Try:
kubectl exec <pod_name> -n <namespace_name> -- /usr/bin/mysqldump -u <db_user> --password=<db_password> --verbose <db_name> | tee <file_name>
This will instead output to stdout as well as sending to the file. This will guarantee that kubectl will see data coming back and will not disconnect you if this is a no-data problem. The downside to this is that you are getting the entire SQL data being pumped back to you on stdout
The other alternative (and one I would personally recommend), is installing something like screen or tmux or another terminal multiplexer and doing the dump in that so you can disconnect from the pod and not worry about kubectl disconnecting you.
EDIT: Just to clarify, I meant installing the multiplexer inside the pod (e.g. inside the docker image you use to create the pod)