If you’ve explored the performance metrics on Top Shelf Analytics, you may have noticed that we focus on F1 Score rather than overall accuracy when evaluating our NHL shot prediction models.
At first glance, this might seem unusual. Accuracy is one of the most familiar machine learning metrics and is often the first number people look for. However, when predicting player shot totals, accuracy can paint a misleading picture.
The Problem with Accuracy
Imagine we’re trying to predict whether a player will record 4 or more shots on goal.
On a typical NHL slate, most players will not reach that threshold. If a model simply predicts that every player will finish below 4 shots, it could still achieve a surprisingly high accuracy despite providing almost no useful betting insight.
From a statistical perspective, this is a known as an imbalanced classification problem, where one outcome occurs much more frequently than the other.
While that model may achieve respectable overall accuracy, it completely fails at identifying the successful shot predictions that matter most.
The confusion matrix below shows how every prediction is classified. These four outcomes form the foundation of every common classification metric. While accuracy measures the proportion of all correct predictions, Precision, Recall, and F1 Score place greater emphasis on correctly identifying successful shot predictions.
| Metric | Formula | Value |
|---|---|---|
| Accuracy | (TP + TN) / (TP + TN + FP + FN) | 72.4% |
| Precision | TP / (TP + FP) | 21.0% |
| Recall | TP / (TP + FN) | 62.5% |
| F1 Score | (2·TP) / (2·TP + FP + FN) | 31.5% |
These metrics are all calculated from the same confusion matrix. The difference lies in which parts of the confusion matrix each metric emphasizes.
What Actually Matters
For prop-style predictions, the objective isn’t to correctly identify every player who falls short of the threshold.
Instead, we’re interested in correctly identifying players who have a higher probability of reaching or exceeding it.
That means we care about two important questions:
- Precision (Quality): When the model selects a player, how often is that prediction correct?
- Recall (Coverage): Of all the players who actually reach the threshold, how many does the model successfully identify?
A model that has excellent precision but extremely low recall may only recommend one or two players each night.
Conversely, a model with very high recall but poor precision might recommend nearly everyone, resulting in many incorrect picks.
Neither approach is particularly useful on its own.
Why F1 Score?
The F1 Score combines precision and recall into a single metric.
Rather than rewarding models that simply predict the most common outcome, the F1 Score measures how effectively a model identifies successful recommendations while limiting incorrect recommendations.
This makes it especially valuable for sports prediction tasks where correctly identifying the positive outcome is more important than simply maximizing overall accuracy.
Instead of asking:
“How accurate is the model overall?”
the F1 Score asks a more practical question:
“How well does the model identify quality betting opportunities?”
For our application, that’s a much more meaningful measure of performance.
But simply choosing F1 Score isn’t enough – we also need to decide what level of historical F1 performance is sufficient before recommendations are published.
Top Shelf Analytics evaluates each shot prediction model across a range of historical F1 Score requirements. As shown below, increasing the minimum F1 requirement generally improves recommendation quality while reducing the number of qualifying opportunities.
The highlighted circles identify the production operating point (minimum F1 Score = 75%) used across all NHL shot prediction models.
As the minimum F1 Score requirement increases, historical recommendation quality generally improves, but fewer predictions qualify for publication. Rather than optimizing each shot threshold independently, Top Shelf Analytics applies a consistent minimum F1 Score of 75%, providing a practical balance between recommendation quality and the size of the qualifying candidate pool across all NHL shot prediction models.
Looking Ahead
No single metric tells the entire story of a machine learning model.
We evaluated additional safeguards, including minimum historical sample sizes and prior positive-prediction support. While both helped identify recommendations with limited historical support, predicted probability provided a simpler and more intuitive measure of confidence while capturing many of the same low-confidence cases.
Throughout Top Shelf Analytics, we use multiple evaluation metrics – including precision, recall, lift, ROC, AUC, calibration, and F1 Score – to evaluate model performance from different perspectives.
For presenting actionable NHL shot predictions, however, we believe the F1 Score provides one of the clearest summaries of historical recommendation quality. Combined with the model’s predicted probability for each game, the F1 Score provides a practical balance between historical recommendation quality and current-game confidence.