A Kafka topic for my table is not created when using the debezium/connect Docker image. Here's how I'm starting the container:

docker run -it --rm --name debezium -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my-connect-configs \
-e OFFSET_STORAGE_TOPIC=my-connect-offsets  -e BOOTSTRAP_SERVERS=192.168.56.1:9092 \
-e CONNECT_NAME=my-connector -e CONNECT_CONNECTOR_CLASS=io.debezium.connector.postgresql.PostgresConnector \
-e CONNECT_TOPIC_PREFIX=my-prefix -e CONNECT_DATABASE_HOSTNAME=host.docker.internal -e CONNECT_DATABASE_PORT=5432 \
-e CONNECT_DATABASE_USER=postgres -e CONNECT_DATABASE_PASSWORD=root -e DATABASE_SERVER_NAME=mydb \
-e CONNECT_DATABASE_DBNAME=mydb -e CONNECT_TABLE_INCLUDE_LIST=myschema.my_table -e CONNECT_PLUGIN_NAME=pgoutput \
debezium/connect

I've tried using CONNECT__ instead of CONNECT_, but I get the same results. A topic for the table is not created if I use the API:

curl -H 'Content-Type: application/json' 127.0.0.1:8083/connectors --data '
{
  "name": "prism",  
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector", 
    "topic.prefix": my-connector",
    "database.hostname": "host.docker.internal", 
    "database.port": "5432", 
    "database.user": "postgres", 
    "database.password": "root", 
    "database.server.name": "mydb", 
    "database.dbname" : "mydb", 
    "table.include.list": "myschema.my_table",
    "plugin.name": "pgoutput"
  }
}'

The topics my-connect-configs and my-connect-offsets, specified by CONFIG_STORAGE_TOPIC and OFFSET_STORAGE_TOPIC, are created.

http://localhost:8083/connectors/my-connector/status shows this:

{"name":"my-connector","connector":{"state":"RUNNING","worker_id":"172.17.0.3:8083"},"tasks":[{"id":0,"state":"RUNNING","worker_id":"172.17.0.3:8083"}],"type":"source"}

I was able to create a topic when using bin/connect-standalone.sh instead of the Docker image as per this question.

Automatic topic creation is enabled and I don't see any errors/warnings in the log.

CodePudding user response：

1. Check kafka connect container logs, what msg it shows when it tries to insert the data to kafka cluster.
2. You need to enable auto topic creation in kafka broker config, checkout this doc
3. Make sure that table exist in the db and "table.include.list": "myschema.my_table", is correct. For experimentation you can remove this config temporary.

You can use UI platform created by redpanda team to manage topic, broker and kafka-connect config - here

CodePudding user response：

The issue was that the underlying table didn't have any data, and, therefore, the topic was not created. The topic is created either if the table has data when the connector is started or if rows are added while the connector is running.