> ## Content Index
> Fetch the complete content index at: https://www.ffpurpose.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# What Building Machine Learning Models Has Taught Me About Product Discovery
- URL: https://www.ffpurpose.com/when-the-baseline-wins-building-and-breaking-a-healthcare-ml-model-in-a-week/
- Published: 2026-08-17T21:00:17.000Z
- Updated: 2026-10-13T18:22:12.000Z
- Author: tanner stahl
- Tags: Research, Essay

*A cautionary tale for any PM about to greenlight an AI feature: sometimes the simplest approach works better than the complicated model.* 

I work as a product owner for data science and AI products, where part of my job is figuring out which problems are worth solving with machine learning. Outside of work, I wanted to explore that question through a project of my own.

I chose audiology because patient follow-up seemed like a practical opportunity for better prioritization. Clinics need to decide which patients to contact for follow-up care, but traditional recall systems often rely on fixed schedules rather than individual patient needs. I wanted to see whether a model could use hearing test results and patient information to make those decisions more effectively.

I built a working prototype in about a week, then spent the next few weeks testing whether it actually improved the decision I cared about: which patients should be contacted first.

The results surprised me. Across my evaluations, simply sorting patients by a measurement already available on their hearing tests performed better than the machine learning model I'd built.

I still think the product is worth building. I'm just no longer convinced that its first version needs machine learning. Here's how the experiment changed my thinking.

## **Why a model initially seemed like the right answer**

Audiology practices have a straightforward recall problem: patients get tested, leave the clinic, and may need follow-up later. Most practice management systems handle this with calendar-based recall, where a reminder automatically goes out twelve months after the last appointment. That follow-up window is significant because it can end up delaying care for patients who need it the most. In fact, one longitudinal study [found](https://pubmed.ncbi.nlm.nih.gov/30085938/?ref=ffpurpose.com) that people who need hearing aids typically wait an average of nine years before actually adopting them. This is only worsened by unstructured patient recall systems.

The problem with a calendar-based approach is that a patient with substantial hearing loss and one with nearly normal hearing can land in the same twelve-month recall queue, despite having very different reasons to return. This difference was reflected clearly on every audiogram by a number measurement of hearing acuity.

I wanted to know if I could create a better outreach queue using this data. My hypothesis was that a model combining audiometric and demographic features could rank patients by priority more effectively than a fixed schedule. Using roughly 14,500 adult audiograms from seven cycles of the public National Health and Nutrition Examination Survey (NHANES) dataset, I trained a machine-learning model on age, sex, and several audiometric features. With Claude as a co-developer, I built the data pipeline, feature engineering, training, scoring API, and deployment architecture in about a week of part-time work.

## **Could I just do without the model?** 

Building the prototype showed me that I could train and deploy a model for this problem. It didn't tell me whether a practice would be better off using it. Before taking the idea any further, I wanted to know whether the model could improve patient prioritization enough to justify the added complexity.

I used the National Health and Nutrition Examination Survey (NHANES), a publicly available dataset containing hearing test results and demographic information. It gave me enough patient-level information to explore the idea without needing access to a clinic's private records. I trained the model on older survey cycles and reserved a later cycle for testing, so I could evaluate it on data it hadn't seen during training.

For comparison, I tested several simpler approaches. Logistic regression gave me a less complex statistical model to compare against. Sorting by age tested whether a basic demographic characteristic was sufficient. Random prioritization provided a reference point for performance without a useful ranking method.

The most important comparison was also the simplest: sorting patients by their worse-ear pure tone average (PTA), a measurement that summarizes hearing loss using results from an audiogram. Unlike the other approaches, this was something a practice could potentially implement without building a predictive model.

If the model couldn't outperform that straightforward alternative, I would have a difficult time justifying its additional complexity.

I looked at three measures: how well each method separated higher- from lower-priority patients (AUC), how many genuinely high-priority patients appeared in the top 20% of its list (precision), and how much better that was than random prioritization (lift).

*Table 1: Assessing performance by methodology*

| Method              | AUC   | Precision @ Top 20% | Lift  |
| ------------------- | ----- | ------------------- | ----- |
| Worse-ear PTA sort  | 0.967 | 0.873               | 3.92x |
| XGBoost             | 0.938 | 0.835               | 3.75x |
| Logistic regression | 0.931 | 0.820               | 3.68x |
| Age alone           | 0.651 | 0.413               | 1.86x |
| Random              | 0.516 | 0.243               | 1.09x |

The PTA sort won and looking at the way I had defined the priority label helps to explain why. PTA or Pure tone average is a number calculated from an audiogram that summarizes hearing acuity across several frequencies. It is also closely related to the clinical thresholds used to describe the severity of hearing loss, making it a strong baseline for the priority label I was testing.

The model combined age, demographic information, and multiple audiometric measurements, but much of that information was already captured by PTA. I tested this directly by adding PTA as an input, which allowed the model to perfectly separate high-priority from lower-priority patients.

That showed me how little uncertainty was left in the original prediction problem. All I needed was a current audiogram and a single number printed on it. With that, I already had almost everything I needed to prioritize patients efficiently. If I wanted to justify using machine learning here, I would need to find something the audiogram couldn't already tell me.

## **Where would ML fit in here?**

That result didn't kill the idea, but it meant I'd built the wrong experiment. I tested whether other risk factors, like diabetes, hypertension or noise exposure, could catch patients that severity alone missed. They didn't move the needle. Adding them produced roughly the same top-20% precision as the audiogram features alone.

But now I was left with a more interesting version of the same problem. What if, I tried to estimate who was most likely to have worsened since their last visit? Here the current audiogram stops being the answer, because the system has to project forward from an older measurement, the time elapsed, and whatever else is known about the patient. I didn't have longitudinal data to test this directly, so I simulated older audiograms and compared projection methods.

This time, the model performed better. The average projection error rate went down by about 18% compared to the control approach, i.e. assuming a patient's hearing hadn't changed since the last visit.

But better predictions only mattered if they resulted in the right patients being contacted. So I used both approaches to rank patients and compared the top 20% of each outreach list. The lists were nearly identical. 

The reason for that was fairly simple: differences in hearing loss between patients are usually much bigger than the changes I was projecting within an individual patient. Someone starting with substantial hearing loss would therefore tend to rank ahead of someone with relatively good hearing, even if the model predicted each person's future hearing more accurately.

That changed how I interpreted the 18% improvement. The model was better at estimating future hearing loss, but it wasn't meaningfully changing who actually got contacted first.

There is an important limitation here: I didn't have repeat audiograms from the same patients over time, so I had to simulate how their hearing might change. So, I wouldn't draw any conclusions about the value of hearing loss predictions based on this experiment. But I would use it to justify collecting real longitudinal data and testing whether better predictions actually produce a better outreach queue.

The lesson I will carry into my next ML project is that predictive performance and product value aren't the same thing. A model can improve the metric it was trained to optimize without materially improving its decision output. That's exactly what happened here. For a PM, the evaluation shouldn't stop at model performance, but follow the model's output through to the user or business decision it is intended to improve.

## **What I'd want to know earlier next time**

Next time, I'd identify the strongest simple approach **before** building the model. In this project, that meant asking how well I could prioritize patients using the information the practice already had, without machine learning. 

Sorting patients by their level of hearing loss turned out to work extremely well. That simple approach became my baseline and the model would have to beat it to justify its added complexity. 

Also, comparing against random prioritization or the existing calendar-based process ultimately set the bar too low. Once I compared the model against something the practice could realistically use instead, its value looked very different.

I'd also check whether better model performance actually changes what the product does. The practice doesn't care how precisely the system estimates a hearing measurement, but whether the right patients actually get contacted.

Lastly, I'd decide up front what evidence would actually change the product direction. The severity-sort result changed what I'd ship; the forward-projection result changed what data I needed to collect next, but neither made the original ML model easier to defend. What they did do, however, was make the direction clearer, which is what I actually want out of an evaluation in the first place.

## **What I'd actually ship**

If a practice asked me to deploy this tomorrow, I'd start with severity-based prioritization, as it’s easier to explain, easier to validate clinically, and it beats the ML model I built. At the same time, I'd start collecting the longitudinal data needed to test the harder question: is hearing deterioration between visits predictable enough to actually change the outreach queue? 

I don't know yet whether a model like that would beat the one sorting by hearing-loss severity. It might not, but that’s the point of an additional experiment.

I expected the model to be the interesting part of this project. Instead, I learned much more by finding the simplest thing I could ship now, identifying the harder question worth answering next, and getting there fast enough that I could scrap the original model with a clear conscience.