Suppose you want your .NET application to classify data in some way. Given some set of features as input, what is the best label for it as output? For example:

  • Given an image of a hand-written numeral, pick the best digit that it represents
  • Given an invoice, pick the best product category
  • Given the ingredients in a recipe, pick the best wine recommendation

If you have a suitably large set of examples for which you already know the right answer, a multiclass classification model built with Microsoft's ML.Net package is a good tool to use.

This post assumes that you are already familiar with the basics of such models. If you are not, a number of great tutorials can help you begin:

In this post, I will show you how to dig a little deeper into your ML.NET classifier results to get a set of scores and ranked predictions rather than a single label. You can use this information for error analysis and pipeline tuning, and even solving multi-label problems.

The code and explanations that follow are based on work using Microsoft.ML 1.3.1 in F#, and especially multiclass trainers like OneVersusAll and SdcaMaximumEntropy.

Motivation

Typically, a multiclass classifier model is conceptualized as a function from a set of features -- words, pixels, numeric fields -- to a label. Hence, the usual prediction type looks like this:

[<CLIMutable>]
type Prediction = {
    PredictedLabel: string
}

But each output label is merely the highest-scoring candidate among many possible labels. This might leave you wondering:

  • What is the actual numeric score for a prediction?
  • What are the scores for the other labels?

With a little more work, you can augment your predictions with multiple labels and their scores. This can allow you to, for example:

  • View the top five labels for each input
  • View all labels for each input that have a score of at least 0.20 (* scores are typically values between 0. and 1.)
  • View how "strong" the label prediction is compared to others

Implementation

The steps below describe how to "enhance" predictions from an existing ML.NET multiclass model. For guidance in creating a pipeline and training a model, see the tutorials linked near the top of this post.

No changes to an existing ML.NET pipeline or model will be required for the enhancement. Essentially, our goal is to convert a simple Prediction produced by the model to an EnhancedPrediction that contains all candidate labels, along with a number that indicates how "strongly" each label fits the original input.

1. Add a Score to your prediction type

The first thing to do is get a little more information with the predictions made by your model. Where previously your prediction record type just included PredictedLabel, now it will also include Score.

[<CLIMutable>]
type Prediction = {
    PredictedLabel: string
    Score: Single array
}

In order to populate PredictedLabel with your model's prediction, you needed to include a transformation step in your model's pipeline, like this:

myPipeline.Append(
    mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabel"))

Not so for Score! It will be mapped automatically by ML.NET, so no change to your existing pipeline is needed.

2. Extract SlotNames from your DataViewSchema

A prediction's Score is just an array of numbers. You can find a parallel array of labels corresponding to each such number in your model's output schema. The best way to get an output schema depends on how your existing ML.NET model fits within the application as a whole. For simplicity, we will access the schema from a PredictionEngine.

In the code below, we assume that the 'features type parameter is already bound to some input type, and you already have an ML.NET model in scope as an ITransformer called myModel.

let engine = 
  _ml.Model.CreatePredictionEngine<'features, Prediction>(myModel)
  
let outputSchema: DataViewSchema = engine.OutputSchema

Extracting values from an outputSchema is an awkward process at present, requiring you to use mutable variable references. If this technique is strange to you, see Passing By Reference on docs.microsoft.com.

Here is a function for extracting the array of labels that parallels the Score array from Step 1:

let getScoreLabels (schema: DataViewSchema): string array = 
      let labelsFromColumn c =
          let mutable labelBuffer = new VBuffer<ReadOnlyMemory<char>>()
          scoreColumn.Annotations.GetValue("SlotNames", &labelBuffer)
            
          labelBuffer.DenseValues()
          |> Seq.map (fun value -> value.ToString())
          |> Seq.toArray
            
      schema
      |> Seq.tryFind (fun c -> c.Name = "Score")
      |> function
         | Some scoreColumn -> labelsFromColumn scoreColumn
         | None -> failwith "Column with name \"Score\" was not found."

The "SlotNames" string represents an annotation kind used by ML.NET. It is a "magic string," and it should not vary among implementations.

Also note that the array of labels extracted by the code above is an attribute of the output schema, not an attribute of a particular prediction. That means that it can be reused for enhancing any prediction made by your prediction engine.

3. Combine score labels with scores

With parallel arrays for scores and their corresponding labels in hand, you can now combine them to learn more about your model's predictions.

Let's define a new record type to represent an "enhanced" prediction. It includes an array of all candidate labels, along with their scores.

type EnhancedPrediction {
	PredictedLabel: string
	AllScoredLabels: ScoredLabel
}
and ScoredLabel = (string * Single) array

Converting a Prediction into an EnhancedPrediction is easy in F#, thanks to Array.zip.

let enhance (scoreLabels: string array) (prediction: Prediction) =
        { PredictedLabel = prediction.PredictedLabel
          AllScoredLabels = prediction.Score
                            |> Array.zip scoreLabels }

Instead of a mere label, you now have access to all possible for a given input, and a score for each one.

Conclusion

At the beginning of this post, I suggested some ways in which this additional score and label data could be useful to you, such as error analysis and maybe solving multi-label problems.

Before concluding, I should emphasize: this is a "how" post, not a "you should" post. There are many alternatives for tuning your classification model and picking multiple labels per input.

Still, we at Locai Solutions have benefited from the extra data, and knowing more generally how to interact with data views and schemas in a deeper way can be useful for many ML.NET applications.