Newtonsoft Json.Net - How to conditionally add (or skip) items in an array when deserializing?-CodePudding

I'm looking for a solution for a performance optimisation I'm trying to get in place for some code I've got consuming a financial market orderbook, basically the JSON object I'm getting back has an array type property containing hundreds/thousands of order objects however I'm only interested in the top 10 to 20 of these (needs to be dynamically determined). From a performance perspective I'd prefer to just skip deserializing & adding every item in the array after I've added the ones I need. To clarify it's always the FIRST 10 to 20 items in the array I actually need, everything after this can be excluded.

Is there any way to achieve this in Json.NET? I've been looking at JsonConverters but can't figure it out.

CodePudding user response：

You can create the following JsonConverter<T []> that will skip array entries beyond a certain count:

public class MaxLengthArrayConverter<T> : JsonConverter<T []>
{
    public MaxLengthArrayConverter(int maxLength) => this.MaxLength = maxLength >= 0 ? maxLength : throw new ArgumentException(nameof(maxLength));

    public int MaxLength { get; }

    public override T [] ReadJson(JsonReader reader, Type objectType, T [] existingValue, bool hasExistingValue, JsonSerializer serializer)
    {
        if (reader.MoveToContentAndAssert().TokenType == JsonToken.Null)
            return null;
        reader.AssertTokenType(JsonToken.StartArray);
        var list = new List<T>();
        while (reader.ReadToContentAndAssert().TokenType != JsonToken.EndArray)
        {
            if (list.Count < MaxLength)
                list.Add(serializer.Deserialize<T>(reader));
            else
                reader.Skip();
        }
        return list.ToArray();
    }

    public override bool CanWrite => false;
    public override void WriteJson(JsonWriter writer, T [] value, JsonSerializer serializer) => throw new NotImplementedException();
}

public static partial class JsonExtensions
{
    public static JsonReader AssertTokenType(this JsonReader reader, JsonToken tokenType) => 
        reader.TokenType == tokenType ? reader : throw new JsonSerializationException(string.Format("Unexpected token {0}, expected {1}", reader.TokenType, tokenType));
    
    public static JsonReader ReadToContentAndAssert(this JsonReader reader) =>
        reader.ReadAndAssert().MoveToContentAndAssert();

    public static JsonReader MoveToContentAndAssert(this JsonReader reader)
    {
        if (reader == null)
            throw new ArgumentNullException();
        if (reader.TokenType == JsonToken.None)       // Skip past beginning of stream.
            reader.ReadAndAssert();
        while (reader.TokenType == JsonToken.Comment) // Skip past comments.
            reader.ReadAndAssert();
        return reader;
    }

    public static JsonReader ReadAndAssert(this JsonReader reader)
    {
        if (reader == null)
            throw new ArgumentNullException();
        if (!reader.Read())
            throw new JsonReaderException("Unexpected end of JSON stream.");
        return reader;
    }
}

Then, assuming your financial market orderbook model looks something like this:

public class OrderBook
{
    public Order [] Orders { get; set; }
}

You can deserialize as follows:

int maxLength = 20;  // Or whatever you want.
var settings = new JsonSerializerSettings
{
    Converters = { new MaxLengthArrayConverter<Order>(maxLength) },
};
var model = JsonConvert.DeserializeObject<OrderBook>(json, settings);

Assert.IsTrue(model.Orders.Length <= maxLength);

Notes:

In your question you mention only arrays, but if your model is actually using lists rather than arrays, use the following converter instead:

public class MaxLengthListConverter<T> : JsonConverter<List<T>>
{
    public MaxLengthListConverter(int maxLength) => this.MaxLength = maxLength >= 0 ? maxLength : throw new ArgumentException(nameof(maxLength));

    public int MaxLength { get; }

    public override List<T> ReadJson(JsonReader reader, Type objectType, List<T> existingValue, bool hasExistingValue, JsonSerializer serializer)
    {
        if (reader.MoveToContentAndAssert().TokenType == JsonToken.Null)
            return null;
        reader.AssertTokenType(JsonToken.StartArray);
        existingValue ??= (List<T>)serializer.ContractResolver.ResolveContract(objectType).DefaultCreator();
        existingValue.Clear();
        while (reader.ReadToContentAndAssert().TokenType != JsonToken.EndArray)
        {
            if (existingValue.Count < MaxLength)
                existingValue.Add(serializer.Deserialize<T>(reader));
            else
                reader.Skip();
        }
        return existingValue;
    }

    public override bool CanWrite => false;
    public override void WriteJson(JsonWriter writer, List<T> value, JsonSerializer serializer) => throw new NotImplementedException();
}

This answer assumes that you want all arrays of type T [] in your model to be truncated to some specific length in runtime. If this is not true, and you need different max lengths for different arrays to be individually specified in runtime, you will need a more complex solution, probably involving a custom contract resolver.

Demo fiddle here.