I'm working with a Web API which retrieves data from an Elasticsearch database. The database is populated by an entirely different piece of software which I cannot modify. One of the fields in the data, metatag.description, is supposed to be populated with a single value, but sometimes ends up with an array of values because of errors in the upstream data. (I cannot modify those either, it's a collection of ~200 web sites.)

The inconsistent data structure was previously handled by adding a custom JsonConverter to the relevant member of the model.

After updating the project from ASP.NET Core 2.1 to 3.1, the custom JsonConverter is no longer being called when the object is deserialized from Elasticsearch.

The project file references the Microsoft.AspNetCore.Mvc.NewtonsoftJson package.

<PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="3.1.*" />

Newtonsoft.Json is initialized in ConfigureServices():

services.AddControllers().AddNewtonsoftJson();

A cutdown version of the model looks like this (the full file includes using Newtonsoft.Json;). The model has a single Description field, but sometimes the upstream data source sends an array instead of a single value. (We can't reasonably fix the upstream source.)

public class SiteWideSearchResult
{
    [Text(Name = "title")]
    public string Title { get; set; }

    [Text(Name = "metatag.description")]
    [JsonConverter(typeof(MetadataDescriptionConverter))]
    public string Description { get; set; }
}

The JsonConverter currently looks like this, but none of these lines are executed, nor are any breakpoints reached when set (this full file also includes using Newtonsoft.Json;):

public class MetadataDescriptionConverter : JsonConverter
{
    public override bool CanConvert(Type objectType)
    {
        throw new NotImplementedException();
    }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        throw new NotImplementedException();
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        throw new NotImplementedException();
    }
}

As described above, there are two versions of the structure being deserialized. The JsonConverter is meant to handle both and allow a (potentially altered) version of the data be stored in the model.

Because it matches the model, this version deserializes correctly:

      {
        "_index": "myindex",
        "_type": "doc",
        "_id": "item_id_1",
        "_score": 4.3536105,
        "_source": {
          "metatag.description": "Single description entry",
          "title": "This structure works"
        }
      }

This version is meant to be handled by the custom JsonConverter. As it is, it fails.

      {
        "_index": "myindex",
        "_type": "doc",
        "_id": "item_id_2",
        "_score": 4.3536105,
        "_source": {
          "metatag.description": [
             "First description line",
             "Second description line"
          ],
          "title": "This structure fails"
        }
      }

The error from the second structure is expected:'String Begin Token', actual:'[' which makes sense, but that's what the converter is meant to address. Except, it's not being invoked.

I suspect the System.Text.Json serializer is being used instead of the one from Newtonsoft.Json.

This answer seems to suggest adding Newtonsoft in the ConfigureServices and the using statements isn't sufficient. The answer's terseness however leaves me unclear what needs to change (I think it's Razor markup). I've found no documentation suggesting anything similar for a webapi (Possibly that's an attribute? Which is what I'm already using).

Things I've tried

• As written, the model and converter do work with .Net 2.1, but it occurs to me that the deserializer in the newer version of Nest might stop because Description is decorated with [Text]. I've tried changing it to [Nested] without success.

• Removing the mapping attribute altogether and using [JsonProperty(PropertyName = "metatag.description", ItemConverter = typeof(MetadataDescriptionConverter))]. No difference.

• Using the JsonConverter attribute on the entire SiteWideSearchResult model, but that also doesn't execute the converter. (Nor does it hit any breakpoints in the converter.)

CodePudding user response：

The ASP.NET Core change from Newtonsoft.JSON to the native System.Text.Json.Serialization is a red-herring.

The actual culprit is a change to the version 7.x of the Elasticsearch NEST client. As of version 7.0, NEST no longer uses Newtonsoft.Json internally.

For the specific case of deserializing on when retrieving data from Elasticsearch, it appears the correct approach is to specify a custom serializer as described on that page.

For my particular use case, the solution is to use the JsonNetSerializer by adding sourceSerializer: JsonNetSerializer.Default in the call to the ConnectionSettings constructor.

var connectionSettings =
    new ConnectionSettings(pool, sourceSerializer: JsonNetSerializer.Default);
var client = new ElasticClient(connectionSettings);

On the other hand, if the need for custom mapping extends to index creation time, the custom "Visitor Pattern mapping" may be more appropriate.