I could not find any simple explanation on how the Apache Kafka Connect works practically. For instance, in the official tutorial, it says that we run certain shell scripts under the bin directory and we need to configure some .properties files.

This is clear. I am guessing that we need to clone entire apache kafka source code because those shell files are using a lot of other shell scripts and java classes inside the apache kafka source code.

But where do we place our connector code and in which format? As source code or as a jar file? When we say my connector class is "com.example..." how come the kafka knows where to look for it? Could someone clear these up or point out to some tutorials? Note, I am running everything, including my kubernetes kafka cluster locally and I am not willing to use Confluent.

CodePudding user response：

clone entire apache kafka source code

False. Apache Kafka download (binaries, not source), come with Kafka Connect already included.

Kafka Connect is not unique to Confluent. They (and others) write plugins. You put the code in the plugin.path property value as JARs. Related - Use Confluent Hub without Confluent Platform installation

When we say my connector class is "com.example..." how come the kafka knows where to look for it?

The JVM classpath.

kubernetes kafka cluster locally

Strimzi has kind: KafkaConnect resources to make this easier, including installing Connector plugins.