So I built an app on my home machine and it worked perfectly. Unfortunately moving the configuration of three docker containers where one flask container accesses another postgres server fails. I am accessing the postgres container using psycopg2 using a hostname. The issue has been isolated to the connecting between one docker container to the next as I can access the server itself, the backend works if it doesn't have to access to the server, but it fails when it does. So heres the config I have currently (I am happy to send any config needed but in the interest of not sending an infinite amount Ill start with just docker compose):
Here is my docker compose that works on a intel macbook pro with docker compose v2.0.0 but doesnt work on an ampere oracle vm with docker compose v2.1.1
services:
backend:
build:
context: .
dockerfile: Dockerfile.api
# image: react-flask-app-api
depends_on:
- postgres_real
ports:
- "5000:5000"
links:
- postgres_real:postgres_real
networks:
- backend
client:
build:
context: .
dockerfile: Dockerfile.client
depends_on:
- backend
ports:
- "3000:80"
links:
- backend:backend
networks:
- backend
postgres_real:
container_name: postgres_real
image: postgres:latest
restart: always
environment:
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: ${POSTGRES_DB}
PGDATA: /var/lib/postgresql/data/pgdata
ports:
- "5432:5432"
expose:
- "5432"
networks:
- backend
networks:
backend:
driver: bridge
EDIT: So I would like to simplify my question further. After more testing, curl commands and attempting to forward requests through nginx using the following configuration
# nginx configuration for Docker
server {
listen 80;
server_name _;
root /usr/share/nginx/html;
index index.html;
error_page 500 502 503 504 /50x.html;
location / {
try_files $uri $uri/ =404;
add_header Cache-Control "no-cache";
}
location /static {
expires 1y;
add_header Cache-Control "public";
}
location /api {
proxy_pass http://backend:5000;
}
}
I have found that on a oracle server, the docker subnet does not actually work in that it is not able to make a request from one docker container to another. I am not sure what is needed so here are the probable required configs:
Config of my docker network that works on my personal machine but not on the server
{
"Name": "congress_backend",
"Id": "b484525801f859d3f38b59ea3b74baa6214e12acf3fcdf72e2115ee1a7c7c192",
"Created": "2022-12-24T07:50:54.96602112Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.23.0.0/16",
"Gateway": "172.23.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"188de200ef405c8e1b859c57fcecda64c7b951ca38cd17d92db301b2b63677a7": {
"Name": "congress-client-1",
"EndpointID": "a170d1a21231a17c3c367e10dc0daaf56281bb9051161cae67e6e696c9f44ade",
"MacAddress": "02:42:ac:17:00:04",
"IPv4Address": "172.23.0.4/16",
"IPv6Address": ""
},
"5fae87555e453a2d6e6d2d080d1c128932a90dc2e71c75b31a82b1a365713497": {
"Name": "congress-backend-1",
"EndpointID": "63801907b9c92b6ac318bbe6cf69e6b08a5f0b82e9fcb1590977116fce8f4690",
"MacAddress": "02:42:ac:17:00:03",
"IPv4Address": "172.23.0.3/16",
"IPv6Address": ""
},
"73f9ece1a79e94e7442d01823f181e244a952f2bf3fde2e205e4c95a8aa74429": {
"Name": "postgres_real",
"EndpointID": "3d074029759e069f66519a0f3308d50801d1e1c09a48b1fa88870533980f6e45",
"MacAddress": "02:42:ac:17:00:02",
"IPv4Address": "172.23.0.2/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "backend",
"com.docker.compose.project": "congress",
"com.docker.compose.version": "2.1.1"
}
}
And here is my Iptables config for IPV4 rulesv4:
# CLOUD_IMG: This file was created/modified by the Cloud Image build process
# iptables configuration for Oracle Cloud Infrastructure
# See the Oracle-Provided Images section in the Oracle Cloud Infrastructure
# documentation for security impact of modifying or removing these rule
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [463:49013]
:InstanceServices - [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p udp --sport 123 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 51820 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5000 -j ACCEPT
-A INPUT -i docker0 -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i wg0 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -d 169.254.0.0/16 -j InstanceServices
-A InstanceServices -d 169.254.0.2/32 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.2.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.4.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.5.0/24 -p tcp -m owner --uid-owner 0 -m tcp --dport 3260 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.2/32 -p tcp -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 53 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p tcp -m tcp --dport 53 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.3/32 -p tcp -m owner --uid-owner 0 -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.4/32 -p tcp -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 67 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp -m udp --dport 69 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.169.254/32 -p udp --dport 123 -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j ACCEPT
-A InstanceServices -d 169.254.0.0/16 -p tcp -m tcp -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j REJECT --reject-with tcp-reset
-A InstanceServices -d 169.254.0.0/16 -p udp -m udp -m comment --comment "See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule" -j REJECT --reject-with icmp-port-unreachable
COMMIT
My application is configured using a flask application with something like this
@app.route('/api/bill/<bill_slug>')
def bill_data(bill_slug):
data = get_bill_data(bill_slug.upper())
return jsonify(data)
# get bill data uses psycopg2 like this:
conn = psycopg2.connect(
host=os.environ.get('DB_HOST'),
database=os.environ.get('POSTGRES_DB'),
user=os.environ.get('POSTGRES_USER'),
password=os.environ.get('POSTGRES_PASSWORD')
)
# where DB_HOST is in this format postgresql://user:password@postgres_real:5432/database_name
if I try a curl command on something that just returns the time it works but if I try something similar like
curl 0.0.0.0:5000/api/bill/Hello
it returns with
curl: (52) Empty reply from server
internally in docker it errors as
[2022-12-24 16:19:37 0000] [8] [CRITICAL] WORKER TIMEOUT (pid:11)
[2022-12-24 16:19:38 0000] [8] [WARNING] Worker with pid 11 was terminated due to signal 9
[2022-12-24 16:19:38 0000] [12] [INFO] Booting worker with pid: 12
CodePudding user response：
If anyone is wondering how this is solved, I recommend checking out your iptables configuration and seeing if there is a legacy configuration as well. In my case, the legacy iptables configuration blocked all forwarding. After turning this off, docker worked perfectly.