> ## Content Index
> Fetch the complete content index at: https://www.ffpurpose.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Evaluating A Viral AI Model: Build vs. Buy
- URL: https://www.ffpurpose.com/what-evaluating-a-viral-ai-model-taught-me-about-build-vs-buy/
- Published: 2026-10-05T21:27:51.000Z
- Updated: 2026-10-13T18:21:43.000Z
- Author: tanner stahl
- Tags: Essay

A few weeks ago a new model called Jev launched, and within days it was being discussed across the industry, including inside our own organization. We spent the next few weeks evaluating it.

What started as a question about whether to adopt Jev turned into something different. I think Jev is real progress, but I'm no longer convinced it's as new as it looks, and the evaluation changed how I think about where differentiation actually comes from. Here's what I learned along the way.

## **Why Jev caught on so quickly**

TypeSafe AI opened early access to Jev on September 15\. Within days it had coverage in[ Forbes](https://www.forbes.com/sites/the-prompt/2026/09/15/this-200-million-startup-wants-to-fix-ais-overconfidence-problem/?ref=ffpurpose.com),[ TechCrunch](https://techcrunch.com/2026/09/18/a-new-kind-of-ai-model-from-a-chatgpt-inventor-is-thrilling-developers/?ref=ffpurpose.com),[ The Register](https://www.theregister.com/ai-and-ml/2026/09/16/typesafe-ai-debuts-model-for-machines-that-plays-doom/5296711?ref=ffpurpose.com) and[ SiliconANGLE](https://siliconangle.com/2026/09/16/typesafe-ai-exits-stealth-with-40m-to-build-ai-for-use-by-software/?ref=ffpurpose.com), and demand was heavy enough to cause temporary outages.

What makes this unusual is what Jev can't do. It doesn't write, summarize or chat. You give it some text plus a question with a fixed set of possible answers, and it returns a probability for each one. Ask whether a support ticket is about billing, a technical issue or sales, and it might reply: billing 8%, technical 85%, sales 7%.

The founder's background certainly helped. CEO Diogo Almeida spent roughly four years at OpenAI working on ChatGPT and the human-feedback training technique (RLHF) behind it, and the company raised a $40 million seed round at a reported $200 million valuation. But I think developers were responding to something more practical: Jev named a problem almost everyone building with large language models (LLMs) has run into.

A lot of what companies send to LLMs today isn't really generation. It's decisions: route this ticket, flag this message, check whether this record meets a criterion. Teams pay frontier-model prices, wait several seconds, and then write code to pull a one-word answer out of a paragraph of text. TypeSafe calls Jev's replacement for this "smart if-statements," and pricing it at[ $0.042 per million input tokens, with output free](https://docs.typesafe.ai/models?ref=ffpurpose.com), made the pitch hard to ignore.

## **What is Jev, really?**

TypeSafe has shared less than the headlines suggest. According to its[ launch post](https://typesafe.ai/blog/introducing-system-one-models-and-jev?ref=ffpurpose.com) and documentation, Jev is a transformer model trained only on synthetic data, using a method the company calls Reinforcement Learning for Calibrated Decisions. The goal is probabilities you can trust: when Jev says 80%, it should be right about 80% of the time. The model weights are proprietary, and there's no technical paper.

The performance claims also deserve a closer look. TypeSafe advertises Jev as 193.6 times faster and 444.6 times cheaper than frontier models, but those figures come from four workflows designed by its own team, and the company itself describes them as "on the higher end of real world gains." As[ TechStock² pointed out](https://ts2.tech/en/typesafe-ai-raises-40-million-for-jev-but-its-445x-cost-claim-is-still-self-tested/?ref=ffpurpose.com), Jev was scored against the average answers of large LLMs rather than against real-world outcomes. In other words, the benchmark measures how cheaply Jev can reproduce what frontier models would say, which isn't the same as measuring whether its decisions are correct.

Two early customers have shared encouraging results. Vercel reported that Jev was 5 to 18 times faster than an OpenAI model for safety classification, and Bryo AI found it 10 to 20 times cheaper than Gemini for email classification. Neither has published how they tested it, and at the time of writing I couldn't find any independent benchmark.

One line in Jev's documentation ended up mattering more to our evaluation than any of these numbers: "Jev is not fine-tuned or LoRA-adapted with customer data." I'll come back to why.

## **Is any of this actually new?**

I went into this evaluation believing Jev's core capabilities had existed since around 2016\. That turned out to be too strong, and figuring out why taught me more than the original claim would have.

Most of the building blocks are well established:

**Table 1: The research behind Jev-style decision models**

| Year | Development                                                                          | What it made possible                                                |
| ---- | ------------------------------------------------------------------------------------ | -------------------------------------------------------------------- |
| 2015 | [Knowledge distillation](https://arxiv.org/abs/1503.02531?ref=ffpurpose.com)         | Training a small, fast model to imitate a large, slow one            |
| 2016 | [fastText](https://arxiv.org/abs/1607.01759?ref=ffpurpose.com)                       | Text classification in milliseconds on ordinary hardware             |
| 2016 | [Data programming (Snorkel)](https://arxiv.org/abs/1605.07723?ref=ffpurpose.com)     | Building training sets without labeling everything by hand           |
| 2017 | [Calibration of neural networks](https://arxiv.org/abs/1706.04599?ref=ffpurpose.com) | Showing models are overconfident, and a simple way to fix much of it |
| 2018 | [BERT](https://arxiv.org/abs/1810.04805?ref=ffpurpose.com)                           | Small models that classify text well with modest labeled data        |
| 2019 | [Zero-shot classification](https://arxiv.org/abs/1909.00161?ref=ffpurpose.com)       | Classifying into categories the model was never trained on           |
| 2021 | [LLMs as labelers](https://arxiv.org/abs/2108.13487?ref=ffpurpose.com)               | Using a large model to generate training labels cheaply              |
| 2024 | [ModernBERT](https://arxiv.org/abs/2412.13663?ref=ffpurpose.com)                     | Faster small models that can read much longer inputs                 |

What none of these could do was answer *any* question you typed, at near-frontier quality, without first training a model for that specific task. A 2016 classifier needed its own labeled dataset for every new decision. So the underlying pattern is about ten years old, and the new part is the generality. That's a real contribution if it holds up under independent testing, though a narrower one than "a new class of frontier model," which is how TypeSafe describes it.

There's also an interesting bit of history here. OpenAI's[ GPT-4 technical report](https://arxiv.org/abs/2303.08774?ref=ffpurpose.com) showed that the human-feedback training Almeida helped pioneer made the model's probabilities less reliable. A company built around trustworthy probabilities looks, to me at least, like a response to that side effect, although TypeSafe hasn't framed it that way.

## **How we evaluated it without running it**

We never actually ran Jev. Our evaluation was entirely desk-based: we researched how it was built and what alternatives already existed, then compared what we'd learned against the projects and priorities already on our roadmap.

That may sound thin next to a proper head-to-head test, and for some questions it is. Desk research can't tell you whether Jev's probabilities hold up on your data. What surprised me, however, was how much of the decision we could settle before testing would have been worth the effort.

Understanding how Jev was built told us what kind of tool it was: a general-purpose decision engine, calibrated on synthetic data and not adapted to any customer. That's useful, but it also told us where its probabilities would be least reliable for us, namely the decisions where our own historical outcomes matter most.

Looking at the alternatives changed the question we were asking. Instead of "Jev or nothing," it became "Jev, the LLM calls we already make, a small model we could train ourselves, or an open-source option." Conversations about hot vendors tend to skip this step, because the excitement frames the choice as adopting the new thing or falling behind.

Comparing Jev against our roadmap was where the decision actually got made. A new capability only matters if it makes something you already need to do cheaper, faster or possible. The lesson I'll carry forward is that for a technology that's only weeks old, a structured desk evaluation is the right first step. Jumping straight to a pilot feels like progress, but without first understanding how a product was built, what the cheaper alternatives are and which of your priorities it actually affects, a pilot tends to test the vendor's demo rather than your problem.

## **Should you build your own?**

You can build something Jev-like for a single, specific decision using open models and well-documented techniques. What you can't easily build is Jev itself: one model that handles any question.

In practice, a prototype would look something like this. Use an open-source LLM to label a few thousand examples, have people label a smaller set to check accuracy, train a small, fast classification model on those labels, and then tune its probabilities against real outcomes. Low-confidence cases get routed to a larger model or a human reviewer. The result runs in milliseconds, costs almost nothing per decision and stays inside your own security boundary.

The real costs show up in production. Someone has to own the labeling process, monitor the model as data changes, retrain it, and update it when the business adds a new answer option. Multiply that by every decision you want to automate, and Jev's generality starts to look like what you're really paying for.

The best-known research on this also cuts against my own instincts. MIT NANDA's 2025 *GenAI Divide* report[ found](https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/?ref=ffpurpose.com) that AI tools bought from specialized vendors succeeded about 67% of the time, while internal builds succeeded only about a third as often. The study's methodology has been criticized and its definition of success is loose, but leaving it out would be cherry-picking.

That's why I've come to think "build or buy" is the wrong question. The more useful one is: which decisions are worth building for?

## **Where differentiation still comes from**

This is where that line from Jev's documentation comes back in.

A probability is only "calibrated" relative to a particular population.[ Research by Ovadia et al.](https://arxiv.org/abs/1906.02530?ref=ffpurpose.com) showed that calibration degrades when the data a model sees in production drifts away from what it was trained on. A model can be well calibrated in one setting and confidently wrong in another.

Jev is calibrated on synthetic data, benchmarked against what other LLMs say and, by design, not adapted to any customer's data. For routing support tickets, that's probably fine. But for decisions where a probability drives money or risk, like flagging a likely fraudulent claim, checking whether a patient qualifies for a clinical trial or pricing an insurance policy, being calibrated to someone else's world is a structural limitation.

That's where I think differentiation survives, although it's narrower than the usual talk about "data moats." Andreessen Horowitz partners Martin Casado and Peter Lauten[ argued in 2019](https://a16z.com/the-empty-promise-of-data-moats/?ref=ffpurpose.com) that most data moats are weaker than founders believe, and I agree when it comes to input data like documents and records, which often look much like everyone else's.

What's much harder to replicate is outcome data, i.e. a record of what actually happened after a decision was made: which claims were paid, which patients enrolled, which policies turned out to be profitable. These can't be generated synthetically, and a general model can't train on yours. They're what turn a generic probability into one you can rely on for your own customers.

My industry has been doing this for a long time. Actuarial work is largely about turning messy historical outcomes into probabilities you can price against. A model like Jev doesn't threaten that. If anything, it makes that outcome history more valuable, because it offers a cheap, general starting point that only your own data can calibrate.

## **A simple way to decide**

Two questions do most of the work: how often you make the decision, and whether you have records of how past decisions turned out.

**Table 2: Choosing between an LLM, a vendor model and your own**

|                 | No outcome records                                                                    | Your own outcome records                                                                                           |
| --------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| **Low volume**  | Use a frontier LLM. Neither a vendor model nor a custom one will pay back the effort. | Buy or use an LLM, but test any vendor's probabilities against your outcomes first.                                |
| **High volume** | A strong case for a general model like Jev, once it's been tested on your data.       | Build. This is where your own model becomes a differentiator, and where a vendor's general calibration is weakest. |

## **What I'd want to know before signing**

If a vendor like this came to us tomorrow, here's what I'd ask for, regardless of how much attention it was getting:

- **Results on our data**, compared with the cheapest reasonable alternative, whether that's a simple statistical model or the LLM we already use. If it doesn't beat that, the vendor's benchmark doesn't matter much.
- **Evidence of calibration on our data.** If the product is trustworthy probabilities, I'd want to see them measured on a sample of our own cases. TypeSafe's launch materials don't include this.
- **How it holds up over time.** Run it on last quarter's data and this quarter's, and see how much the results shift.
- **The cost of an error, not just the error rate.** A 2% error rate on ticket routing and a 2% error rate on claim triage are very different products.
- **How hard it would be to leave.** If prices change or the vendor is acquired, how easily could we swap in our own model?

There's an important limitation to all of this. Jev is only three weeks old, I haven't run it myself, and every performance figure here comes from TypeSafe or its customers. It's entirely possible its real-world calibration is excellent and my concerns about synthetic training are overstated. I also haven't measured how close a homegrown model would get to Jev on the same task, so I wouldn't treat any of this as a verdict on Jev itself.

I would treat it as a reason to slow down, though. I expected the interesting part of this evaluation to be the model. Instead, I learned more by asking what it was actually built from, what the cheaper alternatives were, and which of our own assets no vendor could replicate.