Same exact docker image, services within running container have two different behaviors-CodePudding

My entrypoint.sh file looks like this:

root@e7ebb78ed443:/# cat /opt/entrypoint.sh
#!/bin/bash
cd /
systemctl start mosquitto.service
systemctl start notus-scanner
systemctl start postgresql@14-main
systemctl start [email protected]
systemctl start ospd-openvas
systemctl start gvmd
systemctl start gsad
exec "$@"

It only works if I manually start the gvmd service using systemctl start gvmd, on top of what already ran when executing the entrypoint.sh file.

However, I notice on one system, the gvmd service starts just fine, whereas on the next system, the gvmd service doesn't start because it has a fatal issue with PostgreSQL.

When inspecting the log files of gvmd on the second system, I'm seeing:

md   main:MESSAGE:2023-01-12 21h05.46 utc:46:    Greenbone Vulnerability Manager version 22.4.0~dev1 (DB revision 250)
md manage:WARNING:2023-01-12 21h05.46 utc:47: sql_open: PQconnectPoll failed
md manage:WARNING:2023-01-12 21h05.46 utc:47: sql_open: PQerrorMessage (conn): connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is starting up
md manage:WARNING:2023-01-12 21h05.46 utc:47: init_manage_open_db: sql_open failed

However, the first system shows me this:

md   main:MESSAGE:2023-01-12 18h06.42 utc:52:    Greenbone Vulnerability Manager version 22.4.0~dev1 (DB revision 250)
md manage:   INFO:2023-01-12 18h06.46 UTC:72: osp_scanner_feed_version: No feed version available yet. OSPd OpenVAS is still starting
md manage:   INFO:2023-01-12 18h06.56 UTC:80: osp_scanner_feed_version: No feed version available yet. OSPd OpenVAS is still starting
md manage:   INFO:2023-01-12 18h07.06 UTC:84: osp_scanner_feed_version: No feed version available yet. OSPd OpenVAS is still starting

and the service runs just fine.

The interesting thing here is that both systems are running Ubuntu 22.04.1 LTS and Docker version 20.10.21. The images are exactly the same and the host OS are the same, with the same services/ports exposed.

I'm running docker in the following manner on both systems:

docker run --privileged --rm --name openvas -ti <my_image> /bin/bash

The sha256 of the image is exactly the same on both systems, so I'm not even sure where to begin troubleshooting. Considering the purpose of docker is to have a consistent image across multiple platforms, I'm a bit torn on how to begin troubleshooting/resolving this.

Any guidance would be greatly appreciated.

CodePudding user response：

Note the relevant error message is

md manage:WARNING:2023-01-12 21h05.46 utc:47: sql_open: PQerrorMessage (conn): connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is starting up

I.e., gvmd can not connect to PostgreSQL because the database is still starting up and not ready for connections, yet.

This also gives a simple explanation why this can be working on one host but not on another host with the exact same image etc.: it is a race condition!

It may be that on the first system PostgreSQL is starting up faster - due to a fewer data in the database which is faster to initialize or due to a faster harddrive - and thus is ready as soon as gvmd tries to connect. Or the other way around: gvmd is starting up slower giving PostgreSQL more time to initialize...

A quick - and quite dirty! - solution could be to just additional waiting time in your entrypoint script:

#!/bin/bash
cd /
systemctl start mosquitto.service
systemctl start notus-scanner
systemctl start postgresql@14-main
systemctl start [email protected]
systemctl start ospd-openvas

# wait 10s to allow databases to be ready
sleep 10

systemctl start gvmd
systemctl start gsad
exec "$@"

A better way, though, would be to wait/detect when the database is ready to accept connections. Even better this would be done by adjusted systemd unit files automatically for you, such that systemctl start gvmd will wait until the database is available.