How to access Kubernetes Service via API in Cloud Composer-CodePudding

I am trying to send an API request to a Kubernetes Service using Cloud Composer. I have the following snippet:

def fetchToken():
    ...


def start_application():
keycloak_token = fetchToken()
url = "http://<name>.svc.cluster.local/tasks"

with open("/home/airflow/gcs/dags/src/somefile.json") as f:
    d = json.load(f)

response = requests.put(
    url,
    headers={
        "Authorization": f"Bearer {keycloak_token}",
        "Content-type": "application/json",
    },
    data=json.dumps(d),
)

However, this gives me the following error:

(HTTPConnectionPool(host='<name>.svc.cluster.local', port=80): Max retries exceeded with url: /tasks:8080 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f03e0d97d60>: Failed to establish a new connection: [Errno -2] Name or service not known')); 25941)

Seems like I cannot access as the URL is "svc.cluster.local". Any idea how to access the endpoint?

Thanks!

CodePudding user response：

Cloud composer is a google service independent from GKE, if you don't want to expose your api to public, you can create an internal load balancer to expose the service in the vpc, so by using the same vpc on cloud composer and GKE, your cloud composer dags could access the API in GKE.

There are many other options and tools to achieve this.