Why do I have a slow performance when fetching product prediction with MLContext?-CodePudding

I am building a recommendation engine for products, but I am getting a very slow performance then I trying to fetch and map the product prediction. I am guessing that is the Enumerable.Range(1, 9999999) Is there any other way to do this?

public IEnumerable<(int ProductID, float Score)> GetPredictionTopBoughtProduct(
        PredictionEngine<ProductEntry, CoPurchasePrediction> predictionEngine,
        string activeCode,
        int recommendationCount)
    {

        return (from m in Enumerable.Range(1, 999999)
            let p = predictionEngine.Predict(
                new ProductEntry()
                {
                    ProductId = Convert.ToUInt32(activeCode),
                    CoPurchaseProductId = (uint)m
                })
            orderby p.Score descending
            select (ProductID: m, Score: p.Score)).Take(recommendationCount);
    }

This is the models

 public class ProductEntry
{
    [ColumnName("Label")]
    public float Label { get; set; }

    [KeyType(count: 9999999)]
    public uint ProductId { get; set; }

    [KeyType(count: 9999999)]
    public uint CoPurchaseProductId { get; set; }
}

public class CoPurchasePrediction
{
    public float Score { get; set; }
}

CodePudding user response：

Sorry but i am not familiar with the predict engine you are using here.

But i assume you are using LINQ to SQL with an or mapper like entity framework.

a common issue here would be that "predictionEngine.Predict" cannot be translated into SQL. That would result into single queries of each entity in the enumerable.

If that is the issue, you need to either change the statement, so it can be translated by LINQ to SQL or query all entities first and process them after the materialization. (single query of many is way faster then many queries of single entities).