> ## Content Index
> Fetch the complete content index at: https://www.ffpurpose.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# From 36% to 79%: Testing LLMs for Clinical Trial Matching
- URL: https://www.ffpurpose.com/from-36-to-79-testing-llms-for-clinical-trial-matching/
- Published: 2026-09-08T17:58:25.000Z
- Updated: 2026-09-08T23:00:10.000Z
- Author: tanner stahl
- Tags: Research

Twenty-five patient-trial pairs went through this pipeline. Not one came back eligible.

That includes the patient I built specifically to pass.

The reason wasn't a bug. Every trial had at least one criterion the system couldn't resolve, usually something requiring investigator judgment. The pipeline could confidently rule patients out, but it could never confidently rule them in.

That wasn't the result I was looking for. I built this experiment to answer a simpler question: does using an LLM to read free-text eligibility criteria actually improve on the structured fields ClinicalTrials.gov already provides?

### The experiment

*This is an engineering experiment, not a clinical validation study.*

ClinicalTrials.gov gives you age, sex, and condition as structured fields for every trial, essentially for free. The eligibility criteria that actually decide whether a patient qualifies live in a free-text block: HbA1c thresholds, required medications, pregnancy exclusions, prior-therapy requirements. A human evaluating a patient against these trials still has to reason through that free text. The question I wanted answered was whether reading it with an LLM is accurate enough to justify the cost.

I built a small deterministic rules engine by hand, encoding the checkable criteria from five real Type 2 diabetes drug trials, and used it as a reference standard before writing a single line of LLM code. If I had used an LLM to grade an LLM's extraction, any accuracy number that came out the other end would tell me nothing except that the model agrees with itself. 

Five trials, narrowed from an initial pool of thirty. Even that narrowing required manual review, since condition-name search returned several trials structurally incompatible with the patient population I wanted to test. Five synthetic patients, each designed to test a specific gap: one clean control, one demographic mismatch, one passing every structured field while carrying a free-text exclusion, one inside one trial's recency window and outside another's, one never eligible for any trial to begin with. All data is public, no PHI, everything reproducible from the repo at [github.com/tannerstahl1996/clinical-trial-matching](https://github.com/tannerstahl1996/clinical-trial-matching?ref=ffpurpose.com).

![](https://storage.ghost.io/c/e7/13/e713053b-8704-4cf5-bc37-27c031e3b55e/content/images/2026/09/architecture-diagram.svg)

### The cheap baseline

The structured-field baseline checks two things: the patient's age falls in the trial's age range, and their sex matches the trial's sex eligibility. Both facts are already parsed into structured fields by the API. No LLM call, no cost.

This baseline exists because it is what a team on a budget and time constraint would build. It is also the floor any other approach has to beat.

It caught five of the fourteen pairs where a real disqualifying criterion actually exists. Every one of those five catches was the same demographic mismatch: a sixteen-year-old with Type 1 diabetes fails the age check on every adult T2DM trial in the set.

### Adding the LLM

The extraction step sends each trial's free-text criteria block to Claude Sonnet 4.6 with a structured prompt asking for age range, BMI range, HbA1c range, required medications, excluded medications, and whether Type 1 diabetes or pregnancy are excluded. The model returns JSON. Patients get evaluated against that JSON using the same range-check and substring-match logic the rules engine uses internally.

Extracting all five trials cost $0.026\. Because extraction happens once per trial and can be reused across patients, the effective cost across this experiment was about $0.001 per patient-trial check.

The LLM caught eleven of the fourteen real disqualifications. The layered pipeline, which uses the free baseline first and only falls through to the extracted criteria when the baseline can't reject the pair, hit the same seventy-nine percent while resolving twenty percent of all pairs for free.

![](https://storage.ghost.io/c/e7/13/e713053b-8704-4cf5-bc37-27c031e3b55e/content/images/2026/09/results-table.svg)

On paper, that looked like the answer: 36% recall became 79%, while 20% of pairs never needed the LLM.

### The three misses

**Case one.** A sixty-year-old patient's most recent trial participation was two hundred days ago. One trial in the set excludes anyone who has participated in a clinical trial in the past three hundred sixty-five days. Two hundred is well inside that window. The rules engine correctly rejected the pair. The LLM did not.

The model wasn't wrong about anything it read. The extraction schema simply didn't ask about trial recency. The prompt requests age, BMI, HbA1c, medications, diabetes type, pregnancy. Nothing about prior trial participation. Whatever the model extracted, it couldn't have caught this exclusion.

**Case two.** A newly diagnosed patient had never failed a prior therapy. One trial requires patients who have already failed prior therapy at stable doses. Same story: the criterion is stated clearly in the free text, the rules engine catches it, the extraction schema doesn't ask for it.

**Case three.** A patient's recent medication list includes semaglutide. One trial excludes "GLP-1 analogues (past 3 months)." Semaglutide is a GLP-1 analogue. The extraction correctly pulled "GLP-1 analogues" into the excluded-medications list. The matching logic then tried to check whether the patient had taken any excluded medication by looking for substring matches between the patient's medication names and the extracted exclusions. The string "semaglutide" doesn't appear inside the string "GLP-1 analogues," and vice versa. No match.

![](https://storage.ghost.io/c/e7/13/e713053b-8704-4cf5-bc37-27c031e3b55e/content/images/2026/09/worked-example.svg)

All three misses happened downstream of the model reading the criteria. Two criteria had nowhere to go because I hadn't included them in the extraction schema. The third was extracted correctly, but my matching logic didn't know that semaglutide is a GLP-1 analogue.

The model had mostly solved the reading problem. I had not solved the representation problem.

### Same input, different answer

I reran extraction eight times per trial to see whether model variance created another failure mode. Most fields were stable but one in particular was not.

The same required medication appeared as "SGLT2i," "SGLT2 inhibitor," and "sodium-glucose cotransporter 2 inhibitor (SGLT2i)." Because my matcher used substring containment, one phrasing failed to match the patient's medication record. Three of eight identical extraction runs therefore produced a different downstream eligibility verdict.

The problem wasn't really that the model used different words. A drug ontology would recognize all three as the same class. The failure came from combining nondeterministic extraction with brittle downstream matching.

![](https://storage.ghost.io/c/e7/13/e713053b-8704-4cf5-bc37-27c031e3b55e/content/images/2026/09/failure-taxonomy.svg)

### What I would actually ship

I would not ship this as an "AI eligibility matcher."

I would ship it as an exclusion and triage system. Reject obvious mismatches with structured fields. Run the remaining trials through extracted free-text criteria. For anything left, show the reviewer exactly what remains unresolved and why: missing data, ambiguous criteria, or mandatory investigator judgment.

The output isn't "this patient is eligible." It's "these 40 trials became these 6, and here are the four questions a human still needs to answer."

Three engineering changes would materially improve what this experiment produced:

- Extend the extraction schema to include the fields it currently omits (trial recency, prior-therapy history, OR-conditions).
- Map medications through a real drug ontology like RxNorm so semaglutide and GLP-1 analogues are recognized as related.
- Reconcile multiple extraction runs or constrain the output vocabulary so nondeterminism can't produce string-mismatched variants of the same fact.

These are all engineering problems, not LLM problems.

### Limitations

This is a small experiment: five synthetic patients against five T2DM drug trials. The 79% recall is a result for this dataset, not a clinical-trial-matching benchmark. I manually reviewed all ten disagreements between the systems against the original trial text and found no errors in the reference implementation, but that review was still done by one person.

More importantly, this experiment measured recall of known exclusions. It did not measure whether the system can accurately determine eligibility. The reference implementation itself only produced ineligible or undertermined verdicts.

Cost estimates are similarly directional, because eligibility-criteria length varies substantially across trials.

Next I'd widen this from five tightly related T2DM trials to 50 trials across multiple therapeutic areas and test whether the exclusion-first architecture survives messier eligibility criteria. If it does, this starts looking less like an experiment and more like a product architecture worth building.