Last week I said I’d start a short series on AI in healthcare IT. Here’s what it covers.
I lead DevOps in healthcare IT. “Which AI model should we use?” sounds like one question. It’s really a dozen: how well it reads our data, what it costs to run, where that data goes, who carries the liability when it’s wrong, and whether we can even trust our own test of it. So I built a private test bench at home to work through them with evidence instead of reputation — two used data-center GPUs running open models fully offline, each answering the same few hundred questions about synthetic patient charts from Synthea, scored automatically against the data. No real records anywhere. Then I ran the same test on cloud-hosted models, frontier included, for comparison.
One question each Tuesday:
- Do medical models actually beat general models at reading a chart? I was surprised.
- Everyone fears a medical AI will fabricate clinical information — a lab value, a finding, a prognosis. That wasn’t the failure that worried me.
- Can you trust your own evaluation? The script grading my models had four bugs of its own.
- An AI that flags every possible error sounds safe — until it raises so many false alarms that people stop trusting the real ones.
- Running a model in production is real engineering. The model is the cheap part.
- Build or rent: what would owning the hardware actually cost next to cloud prices?
- Accuracy is just one factor. Where your data goes, the license terms, who’s liable — how do you weigh them all?
Each Tuesday’s post comes with a fuller write-up here — the method, the numbers, and the caveats.
If you’re evaluating AI for healthcare, which of these would you want answered first?
This kicks off a series I’m posting on LinkedIn — follow along there.