Missing field in avro schema-CodePudding

I'm trying to create schema for avro messages for adding to schema registry in Kafka. Since I get messages from mongodb the structure of messages differs from one message to another. For example, one of the field are present in one message but absent in another. For it I use the following schema:

{
   "name": "my_field"
   "type": ["string", "null"]
}

But if the message doesn't have "my_field" field, the message is discarded. And it is not the behaviour I need. I need the system to skip this message. What should I do to set up it?

CodePudding user response：

To make a field optional in Avro you can add the default attribute to your schema definition.

{
   "name": "my_field"
   "type": ["null", "string"],
   "default": null
}

It is important to specify null first when the type is a union. More information can be found in the Avro documentation

CodePudding user response：

Unclear what is skipping messages. If you are catching deserialization errors, and ignoring them, then that needs to be handled at the consumer side, not in the deserializer.

Avro is strictly structured, therefore your collection(s) should be as well.

If you want to handle any random data, then you will want to use JSON in Kafka, rather than Avro. Otherwise, you need to add default: null fields to the schema.