load an XML file into kafka topic without schema-CodePudding

I'm trying to load a rather large XML file into a kafka topic. I know how to load JSON, AVRO etc. but those are parsed out well in topics because their schemas can be inferred. How do I do this with XML? how can I set a key for the XML messages?

I tried something like this

confluent kafka topic produce test1 --parse-key --delimiter , < NameOfFile.xml

but this just produced each line of the XML file as a new message (just text in value).

The XML file looks something like the following (but much longer):

<Mp ver="3" id="517131">
      <Fields>
        <aexst>N</aexst>
        <chexst>N</chexst>
        <clueexst />
        <ocdexst>N</ocdexst>
        <id>123</id>

I want the message key to say:

123

and the message value to be (just a string of the XML file):

<Mp ver="3" id="517131">
      <Fields>
        <aexst>N</aexst>
        <chexst>N</chexst>
        <clueexst />
        <ocdexst>N</ocdexst>
        <id>123</id>

How do I go about doing this without a connector?

CodePudding user response：

You can try to use kcat, or some other tool, to produce to Kafka instead of using the confluent one.

If you just want to publish one file

kcat -P -b <broker_url> -t <topic> -p <partition_number> -k <key - 123 in your example> file.xml

You'll need to be careful to choose the appropriate partition for your message or just set -1 to send it to a random partition.

If you don't want to hard-code the key in the command or you have multiple files, you can use yq to extract the key from the file programmatically and feed it to the kcat command.