> ## Content Index
> Fetch the complete content index at: https://www.ffpurpose.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# 12,000 UFC Scorecards vs. Machine Learning And Human Judgement
- URL: https://www.ffpurpose.com/what-12-000-ufc-fight-scorecards-taught-me-about-machine-learning-and-human-judgement/
- Published: 2026-10-06T18:26:34.000Z
- Updated: 2026-10-13T18:21:20.000Z
- Author: tanner stahl
- Tags: Research

*How to evaluate a model when the ground truth is a panel of humans who disagree, and sometimes get it wrong.*

---

## Twenty-five confident misses

I trained a model to predict how professional UFC (Ultimate Fighting Championship) fight judges would score a fight from the round-by-round statistics. For fights the model had never seen, it picked the judge's winner 83% of the time. Then I pulled every fight from 2017 to 2023 where the model was at least 80% sure one fighter was ahead on the cards and all three judges had the other fighter ahead.

There were 25 of them. For each, I went through post-fight coverage and the media scorecards that outlets publish after every event. In 10 of the 25, the outside scorers agreed with the model, not the judges.

That number changed how I think about the project. I started out building a model of what judges reward. I ended up with a case study in evaluating a model when the ground truth is a panel of humans who don't always agree with each other, and who are sometimes wrong. That second problem is the one most data scientists actually have. Claims adjudication, chart review, content moderation, clinical labeling: they all hand you labels made by people, and they all tempt you to treat those labels as the truth.

This essay walks through the techniques that worked, the faults I found in the data and in my own first pass, and what I'd carry into a real labeling pipeline. You don't need to know anything about mixed martial arts. The code is public at[ github.com/tannerstahl1996/ufc](https://github.com/tannerstahl1996/ufc?ref=ffpurpose.com).

## Why judging in UFC fights is a good labeling problem

Here's everything you need to know about the sport. A UFC fight runs three or five rounds of five minutes. If nobody is knocked out or submitted, three judges decide who wins. Each judge scores every round independently on a "10-point must" system: the round winner gets 10, the loser usually gets 9, a dominant round can go 10-8, and an even round can be 10-10 (it almost never is). A judge's fight total is the sum of their rounds, and whoever leads on two of the three cards wins.

The official criteria are written down and ranked: effective striking and grappling first, defined by impact, then aggression, then control of the fighting area.

As a labeling setup, this is unusually clean:

- Three independent judges per fight, every time, with a published rubric.
- Aggregated labels. Round-by-round cards exist, but the public statistics source I used only records each judge's fight total, such as 29-28\. You know how a judge scored the fight, not which rounds they gave to whom.
- Rich features. For every round there are counts of strikes landed and attempted, split by target and position, plus takedowns, control time, knockdowns, and submission attempts.
- Outside raters. Journalists publish their own scorecards after every event, which gives you a third opinion when the model and the judges disagree.

I used a public scrape of UFCStats.com: 4,138 decisions from 1995 to October 3rd 2026, about 12,000 judge scorecards. The main analysis uses fights from 2017 onward, when the judging criteria were rewritten. Fights from 2024 on were held back as a test set and evaluated one time at the end of this project.

## The fight scorecards only come in totals

The first technical problem: I wanted round-level weights (how much is a takedown worth in a round?) but only had fight-level labels.

You see the same issue in a lot of real-world data. A claim has an outcome but its line items don't. A patient stay has a readmission flag but the individual orders don't. You have a label for the bag and want to learn something about the items inside it.

**Figure 1\. The model observes round-level statistics but only receives fight-level scorecards**

![](https://storage.ghost.io/c/e7/13/e713053b-8704-4cf5-bc37-27c031e3b55e/content/images/2026/10/data-src-image-41ed1b08-6290-43c9-8c06-f75a102dfffc.png)

*Round statistics are observed for each fighter in each round, but the public data records only each judge’s final fight total.*

The approach that worked was to model rounds and then add up the probability of every way the rounds could produce the observed total.

Each round gets a score from a linear model: the difference between the two fighters' stats, times a set of weights. That score goes through an ordered logistic model with five outcomes: the other fighter wins 10-8, wins 10-9, an even round, this fighter wins 10-9, wins 10-8\. For a three-round fight, there are 5³ = 125 ways the rounds could fall. A small dynamic program over the rounds sums the probability of every combination that produces each possible card total. The model is then fit by maximizing the probability of each judge's actual total.

```python
# D[a, b] = probability the card so far reads (8n + a) - (8n + b)
size = 2 * n + 1
D = np.zeros((size, size)); D[0, 0] = 1.0
for r in range(n):                          # rounds
    new = np.zeros_like(D)
    for c, (da, db) in enumerate(OFFSETS):  # the five round outcomes
        new[da:, db:] += D[:size - da, :size - db] * p[r, c]
    D = new
loglik += np.log(D[a_obs, b_obs])           # the judge's actual card

```

Two things I liked about this. First, it uses every card, including the ones with 10-8 rounds, which matters more than I expected (more on that below). Second, it gives the model's beliefs at every level for free: round win probabilities, the distribution over card totals, and the probability a judge has a given fighter ahead.

## A leak hiding in the row order

The fighter listed first in the data won 61% of decisions.

Fighters listed first aren't better fighters. The source appears to lean toward listing the winner first. Any model with an intercept, or any feature that knows which fighter is "A," would learn that and look better than it is.

The fix was structural. Every feature is fighter A's stat minus fighter B's, the model has no intercept, and the five-outcome thresholds are symmetric. Swap the two fighters and every prediction mirrors exactly, which is now a unit test in the repo.

I'd never have thought to check for this if I hadn't looked at the raw outcome balance first. Most datasets have some version of it. Records get sorted, ordered, or keyed in ways that correlate with the label, and a flexible model will find it.

## One stat gets you 75% accuracy by itself

Before fitting anything, I scored three one-rule baselines on the test set: give each round to whoever landed more significant strikes, more total strikes, or had more control time.

Counting total strikes picked the judge's winner 75.8% of the time. That's a high bar for a "sophisticated" model to clear.

Brackets are 95% intervals from resampling fights, since the three cards from one fight aren't independent. The model is well calibrated too: when it says 70%, judges go that way about 70% of the time (expected calibration error 0.027).

**Figure 2\. Total strikes alone predicted the judges' winner 75.8% of the time**

![](https://storage.ghost.io/c/e7/13/e713053b-8704-4cf5-bc37-27c031e3b55e/content/images/2026/10/data-src-image-9f290cb9-d6f2-41a0-9f01-e7fe22321901.png)

 *Accuracy on the held-out test set. Error bars show 95% intervals from resampling fights.*

**Figure 3\. The model's predicted probabilities were well calibrated**

![](https://storage.ghost.io/c/e7/13/e713053b-8704-4cf5-bc37-27c031e3b55e/content/images/2026/10/data-src-image-a6d2be41-3c9f-434b-a4c4-ab5efdff5cbc.png)

*When the model predicted a 70% chance of winning, the judges favored that fighter at roughly the same rate. Expected calibration error: 0.027.*

## I measured the ceiling wrong

The ceiling row is where I made my most costly mistake.

When your labels come from several annotators who disagree, no model can match all of them. The natural ceiling is inter-judge agreement. My first version used pairwise agreement: how often does one judge's exact round count match another judge's on the same fight? That came out around 66%, and my early prototype was at 58%. The gap looked small, and I described that 66% as a rough human ceiling.

That ceiling answers the wrong question. My model makes one prediction per fight and gets scored against each of three judges. The best any such model can do is predict each fight's most common card, which matches the majority judges and misses the dissenter. Measured that way, the ceiling was about 82%. The real gap was more than three times what I had reported.

The general lesson: match the ceiling to how the model is scored. If the model gives one answer per item and is graded against every judge, the ceiling is majority agreement, not pairwise agreement. If you get this wrong, you'll declare victory early and stop improving the thing that most needs work.

## The data faults, and what each would have cost

Cleaning took longer than modeling, which is typical, and almost every fault would have quietly biased something downstream. Here's the list:

**Table 1\. Data-quality problems found in the UFCStats scrape**

| Fault                                                           | How I found it                                                   | What it would have done                                                                                     |
| --------------------------------------------------------------- | ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| 25 fights listed twice                                          | A check that every fight has exactly three cards failed with six | Double-weighted those fights. My feasibility numbers included them.                                         |
| 22 duplicated round-stat rows, one with conflicting values      | Duplicate-key check on fight, round, fighter                     | Inflated those rounds' stats or silently picked one version                                                 |
| Judge names missing or glued to the score ("Jane Smith28 - 29") | Parser returned 2 or 4 cards for some fights                     | Lost cards, or merged two judges into one, which corrupts any per-judge analysis                            |
| 24 "decisions" that ended early (technical decisions)           | Rounds fought didn't match rounds scheduled                      | Mixed partial fights into full ones                                                                         |
| 5 fights whose cards contradict the announced result            | Checked each card set against "unanimous / split / majority"     | Mislabeled outcomes                                                                                         |
| Point deductions                                                | Found during the miss review, not by any check                   | A deducted point shows up in the card total and looks exactly like a 10-8 round. Undetectable in this data. |
| A stat-recording error                                          | Found during the miss review                                     | Coverage of one fight documented ground strikes that were never recorded.                                   |

*Note: Some issues could be detected programmatically; others were found during manual review.*

## Dropping the awkward labels would have changed the answer

My first prototype took a shortcut. Cards with a 10-8 round break the simple "count the rounds won" logic, so I dropped them: 12% of all cards.

When I built the full five-outcome model and compared, dropping those cards would have nearly doubled the estimated weight of a knockdown, from 1.26 to 2.14 in log-odds terms. The cards I was throwing away were exactly the rounds with the most decisive events.

It was worse for the per-judge analysis. Judges give 10-8s at very different rates: from about 3% to 18% of their cards among judges with 100 or more fights. Dropping those cards would have removed the most generous judges' data at five times the rate of the strictest, which is a selection effect aimed directly at the comparison I wanted to make.

I think this is one of the most common faults in applied work. The hard cases are inconvenient, so they get filtered out "for now," and the filter ends up correlated with the thing being studied. If a category of labels is awkward to model, that's usually a sign it carries information.

## What the model learned

Here are the weights expressed as exchange rates: how many extra landed distance strikes (punches and kicks thrown at range) move a round as much as one unit of each stat.

**Figure 4\. Estimated feature weights expressed relative to one landed distance strike**

![](https://storage.ghost.io/c/e7/13/e713053b-8704-4cf5-bc37-27c031e3b55e/content/images/2026/10/data-src-image-176d1d78-2ebc-4e42-b44c-dcec31172e8f.png)

*Weights are fit on 2017–2021 fights. Values represent the relative change in round score associated with each statistic.*

A few of these line up with the written criteria, which ask judges to reward impact over activity. Big moments dominate. Missed strikes are worth nothing, so attempting a lot doesn't help unless it lands. A stuffed takedown costs you.

Two cautions:

- Control time and ground strikes happen together, so they share credit and their individual numbers shouldn't be over-read.
- These are associations with judges' cards, which is different from what judges consciously score.

The source splits the same strikes three ways (by target, by position, and in total). My first prototype used more than one of those splits at once, which made the weights uninterpretable because the model could trade credit freely between overlapping features. The final feature set never counts a strike twice.

## When the model is confidently wrong, check the labels

Back to the 25 confident misses. To make the predictions honest, every fight got a prediction from a model that never saw it (five-fold cross-fitting over 2017–2023). Of 1,302 fights where all three judges agreed on who was ahead, the model was at least 80% confident on 867 and wrong on 25, about 3% of its confident calls.

I categorized each from post-fight coverage and media scorecards. The rule I used: if media scorers mostly sided with the stats, the judges are the outlier ("questionable scoring"); if media sided with the judges, the stats missed something.

**Figure 5\. At least 10 of 25 confident model disagreements looked more like label noise than model error**

![](https://storage.ghost.io/c/e7/13/e713053b-8704-4cf5-bc37-27c031e3b55e/content/images/2026/10/data-src-image-f07588f6-3c60-433c-a8bf-df66fb5ab88a.png)

Two things stand out. First, at least 10 of the 25 "errors" look more like label noise than model error. That doesn't make the judges wrong in any absolute sense, since media scorers are humans too. It does mean the 94% ceiling is a ceiling on agreeing with judges, which is a different thing from being right. In a labeling pipeline, a model's confident disagreements are some of the most valuable items to send back for review, because a meaningful share of them are mislabeled.

Second, one fighter shows up four times. Bobby Green had ten decisions in this period and lost five. In four of those five losses, the model had him clearly ahead and outside scorers mostly agreed. I can't tell from this data whether that's about his style (lots of feints and volume without visible damage) or about how judges perceive him. It's a reminder that label noise isn't always random. Sometimes it clusters on specific items, and averages hide that.

## Judges have styles, but fewer than you'd think

With three judges on every fight, you can ask whether individual judges score differently. The clean comparison holds the round stats fixed and asks how each judge's cards deviate from the shared model. For each judge with at least 100 cards (14 judges, 2017–2026) I estimated three deviations: how strongly their cards respond to striking stats, to grappling stats, and how readily they award 10-8s. Each deviation is pulled toward zero unless the evidence is strong, the same idea as a random effect, and I corrected for testing 14 judges at once (Benjamini–Hochberg).

Three of 14 differ from the average judge after correction. The strongest weights grappling 42% more than average \[+23%, +60%\] with no change on striking, a clear style. Another tracks the recorded stats more closely than average on both striking and grappling, and is stricter about 10-8s. I'd read that as fewer surprises rather than a preference. Several others looked different before the correction and didn't survive it.

Judges are anonymized in everything I publish. And there's a limit I can't get around: a judge who appears to "favor grappling" might be reacting to damage on the ground that the statistics don't record. With only stats as features, a style preference and a blind spot in the data look identical.

## Write your prediction down before the test

Reading through the misses, leg kicks kept coming up. In several, the fighter the model favored had built their striking edge on leg kicks and lost. I checked across all confident calls: in misses, 24% of the favored fighter's significant strikes were leg strikes, versus 16% in correct calls (difference +8 points, interval +2 to +16).

That's a hypothesis I generated by staring at errors, which is exactly how you fool yourself. So before touching the test set, I wrote down two predictions and committed them to the repo with a timestamp: that judges value a leg strike less than a head strike, and that the test-period misses would again lean on leg strikes.

Both held. On 2024–2026 data alone, a leg strike was worth about half a head strike (ratio 0.50, interval 0.33 to 0.63). The test-period misses leaned on leg strikes again (27% vs 15%), though there were only five misses, so that part is weak evidence. A version of the model that splits strikes by target instead of position scored the same overall (83.8%), so I kept the original model and reported the leg-kick discount as a finding about judges, not a model improvement.

The process matters more than this particular result. Post-hoc patterns from error analysis are cheap. Writing the prediction down before the final test is what turns one into evidence.

## What this doesn't show

- The labels are fight totals. Every round-level conclusion comes from the model, not from observed round scores.
- The statistics don't measure damage, which is the top official criterion. A lot of what judges see isn't in the data.
- Only fights that went to a decision are included. Fights that ended in a knockout or submission are a different population.
- The weights are associations. They describe what predicts judges' cards from recorded stats, not what judges consciously reward.
- The before-and-after-2017 comparison (takedowns fell from 3.9 to 2.1 strikes in value; ground strikes rose from 1.0 to 1.4) is consistent with the rules change, but states adopted the new rules at different times and stat-recording practices may also have shifted.
- The manual miss review used media coverage, not full match viewing, and media scorers tend to have their own biases.

## What I'd carry into a real labeling pipeline

This was a personal project, but the problems I observed are also the ones I see in production work with human-labeled data: adjudicated claims, reviewed charts, eligibility determinations. If I were setting up an evaluation on labels like these, I'd want:

1. The strongest boring baseline first. One stat got 75%. Know that number before anyone shows you the 83%.
2. A ceiling that matches how the model is scored. For one prediction per item graded against several judges, that's majority agreement.
3. A check on the raw outcome balance before building any features, to catch ordering and keying leaks.
4. Logged exclusions with reasons, and suspicion of any filter that removes hard cases.
5. A third rater for confident disagreements. Route the model's most confident misses to someone outside the original labeling process. Some will be model errors, and some will be label errors you'd never find otherwise.
6. Annotator-level analysis with shrinkage and multiple-comparison control, so you find real differences without inventing them.
7. Pre-registration for anything learned from error analysis, tested once on held-out data.

I expected the interesting part of this project to be a table of what judges reward. The more useful finding was that a model trained on disagreeing human labels can be a tool for auditing those labels, as long as you're careful about which disagreements you trust.

## References

1. T. Stahl, *ufc: Evaluating a model of UFC judges' scorecards* (2026), GitHub,[ https://github.com/tannerstahl1996/ufc](https://github.com/tannerstahl1996/ufc?ref=ffpurpose.com)
2. Association of Boxing Commissions, *Unified Rules of Mixed Martial Arts* (2019),[ https://www.abcboxing.com/wp-content/uploads/2020/02/unified-rules-mma-2019.pdf](https://www.abcboxing.com/wp-content/uploads/2020/02/unified-rules-mma-2019.pdf?ref=ffpurpose.com)
3. E. Magraken, *ABC Adopts Overhaul to MMA's Unified Rules* (2016), Combat Sports Law,[ https://combatsportslaw.com/2016/08/02/abc-voting-on-unifed-rule-updates/](https://combatsportslaw.com/2016/08/02/abc-voting-on-unifed-rule-updates/?ref=ffpurpose.com)
4. Greco1899, *scrape\_ufc\_stats* (2026), GitHub,[ https://github.com/Greco1899/scrape\_ufc\_stats](https://github.com/Greco1899/scrape%5Fufc%5Fstats?ref=ffpurpose.com) (commit 1ccacc5, data through October 3, 2026)
5. UFC, *UFC Stats* (2026),[ http://ufcstats.com](http://ufcstats.com/?ref=ffpurpose.com)
6. N. Quadrianto, A. Smola, T. Caetano and Q. Le, *Estimating Labels from Label Proportions* (2009), Journal of Machine Learning Research 10: 2349–2374
7. P. McCullagh, *Regression Models for Ordinal Data* (1980), Journal of the Royal Statistical Society, Series B 42(2): 109–142
8. C. Guo, G. Pleiss, Y. Sun and K. Weinberger, *On Calibration of Modern Neural Networks* (2017), Proceedings of the 34th International Conference on Machine Learning (ICML)
9. MMA Decisions, *Media scorecards for UFC decisions* (2026),[ https://mmadecisions.com](https://mmadecisions.com/?ref=ffpurpose.com)
10. Y. Benjamini and Y. Hochberg, *Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing* (1995), Journal of the Royal Statistical Society, Series B 57(1): 289–300
11. T. Stahl, *Pre-registration: final test-set evaluation* (2026), GitHub,[ https://github.com/tannerstahl1996/ufc/blob/main/reports/preregistration\_test.md](https://github.com/tannerstahl1996/ufc/blob/main/reports/preregistration%5Ftest.md?ref=ffpurpose.com)

---

Coding assistance: Claude was used as a coding partner during development of the data pipeline and model evaluation. Data: via a third-party scrape of UFCStats.com; this project is non-commercial and the repository contains no UFC data. Code:[ github.com/tannerstahl1996/ufc](https://github.com/tannerstahl1996/ufc?ref=ffpurpose.com).