Unable to Error creating a kafka message to producer - Expected start-union. Got VALUE_STRING

    {
    "namespace": "de.morris.audit",
    "type": "record",
    "name": "AuditDataChangemorris",
    "fields": [
        {"name": "employeeID",  "type": "string"},
        {"name": "employeeNumber", "type": ["null", "string"], "default": null},
        {"name": "serialNumbers", "type": [ "null", {"type": "array", "items": "string"}]},
        {"name": "correlationId",  "type": "string"},
        {"name": "timestamp",  "type": "long", "logicalType": "timestamp-millis"},
        {"name": "employmentscreening","type":{"type": "enum", "name": "employmentscreening", "symbols": ["NO","YES"]}},
        {"name": "vouchercodes","type": ["null",
            {
                "type": "array",
                "items": {
                    "name": "Vouchercodes",
                    "type": "record",
                    "fields": [
                        {"name": "voucherName","type": ["null","string"], "default": null},
                        {"name": "authocode","type": ["null","string"], "default": null}
                    ]
                }
            }], "default": null}
    ]
}

when i was trying to create a sample data in json format based on the above avsc for kafka consumer i am getting the below error upon testing

{
    "employeeID": "qtete46524",
    "employeeNumber": {
        "string": "custnumber9813"
    },
    "serialNumbers": {
        "type": "array",
        "items": ["serialNumbers3521"]
    },
    "correlationId": "corr-656532443",
    "timestamp": 1476538955719,
    "employmentscreening": "NO",
    "vouchercodes": [
{
        "voucherName": "skygo",
        "authocode": "A238472ASD"
    }
]
}

getting the below error

Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got VALUE_STRING

how to create correct sample data based on the above schema ?

CodePudding user response：

Read the spec

The value of a union is encoded in JSON as follows:

if its type is null, then it is encoded as a JSON null; otherwise it is encoded as a JSON object with one name/value pair whose name is the type’s name and whose value is the recursively encoded value

So, here's the data it expects.

{
    "employeeID": "qtete46524",
    "employeeNumber": {
        "string": "custnumber9813"
    },
    "serialNumbers": {"array": [
        "serialNumbers3521"
    ]},
    "correlationId": "corr-656532443",
    "timestamp": 1476538955719,
    "employmentscreening": "NO",
    "vouchercodes": {"array": [
       {
        "voucherName": {"string": "skygo"},
        "authocode": {"string": "A238472ASD"}
       }
    ]}
}

With this schema

{
  "namespace": "de.morris.audit",
  "type": "record",
  "name": "AuditDataChangemorris",
  "fields": [
    {
      "name": "employeeID",
      "type": "string"
    },
    {
      "name": "employeeNumber",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "serialNumbers",
      "type": [
        "null",
        {
          "type": "array",
          "items": "string"
        }
      ]
    },
    {
      "name": "correlationId",
      "type": "string"
    },
    {
      "name": "timestamp",
      "type": {
        "type": "long",
        "logicalType": "timestamp-millis"
      }
    },
    {
      "name": "employmentscreening",
      "type": {
        "type": "enum",
        "name": "employmentscreening",
        "symbols": [
          "NO",
          "YES"
        ]
      }
    },
    {
      "name": "vouchercodes",
      "type": [
        "null",
        {
          "type": "array",
          "items": {
            "name": "Vouchercodes",
            "type": "record",
            "fields": [
              {
                "name": "voucherName",
                "type": [
                  "null",
                  "string"
                ],
                "default": null
              },
              {
                "name": "authocode",
                "type": [
                  "null",
                  "string"
                ],
                "default": null
              }
            ]
          }
        }
      ],
      "default": null
    }
  ]
}

Here's an example of producing and consuming to Kafka

$ jq -rc < /tmp/data.json | kafka-avro-console-producer --topic foobar --property value.schema="$(jq -rc < /tmp/data.avsc)" --bootstrap-server localhost:9092 --sync
$ kafka-avro-console-consumer --topic foobar --from-beginning --bootstrap-server localhost:9092 | jq
{
  "employeeID": "qtete46524",
  "employeeNumber": {
    "string": "custnumber9813"
  },
  "serialNumbers": {
    "array": [
      "serialNumbers3521"
    ]
  },
  "correlationId": "corr-656532443",
  "timestamp": 1476538955719,
  "employmentscreening": "NO",
  "vouchercodes": {
    "array": [
      {
        "voucherName": {
          "string": "skygo"
        },
        "authocode": {
          "string": "A238472ASD"
        }
      }
    ]
  }
}
^CProcessed a total of 1 messages