Case Study: Zillow's $500M Mistake in Trusting Their Model
When you let the model's make decisions with money, you better have strong guardrails in place.
Open Zillow right now and type in your own address. A number comes up - the Zestimate. It’s Zillow’s guess at what your home is worth, and it is almost certainly a little wrong. By Zillow’s own accounting it misses by about 2% on homes that are actually for sale, and by 7%+ on the ones that aren’t. It’s been doing this since 2006, and it’s never cost anyone anything.
In the autumn of 2021, that same guess, more or less, cost Zillow half a billion dollars and around two thousand jobs.
Not because it got worse. The estimate that caused all this was no better and no worse than the one still sitting on on their website right now.
What changed in between was a single decision: for three years, Zillow let the guess buy the house.
It launched Zillow Offers in 2018, pointed the model at the open market, and started buying real homes with real money on the strength of a number Zillow itself knew was only ever approximate.
The day the guess got a chequebook
For fifteen years the Zestimate did one job: it talked.
It sat on a listing and told you a number, and you were free to believe it, ignore it, or laugh at it. Being wrong had no consequence, because nothing downstream moved when it missed. A wrong Zestimate is just a slightly wrong web page.
Zillow Offers was the decision to give that number a second job.
The idea, which Zillow launched in 2018, was iBuying:
The company would use its own valuation to make instant cash offers on people’s homes
Buy the ones it liked, tidy them up, and sell them on.
It’s one of those things that in theory seems elegant - the company that already estimates every house in America simply acts on its own estimates.
In practice each step quietly turned a soft modelled number into a hard commitment:
an estimate became an offer,
an offer became a purchase,
a purchase became a house Zillow owned, carried, renovated, and had to sell again.
By the time you own the house, the guess has stopped being a guess. Now it is a mortgage, a renovation crew, a property-tax bill, and a bet that the market eight weeks from now will land where the model said it would. The modelled price on the website could be wrong a thousand times a day and cost nothing. The same model with a chequebook had to be right, because every time it was not, Zillow had to eat the difference.
The Cost of the Mistake
You can’t un-buy a house. To undo a bad purchase you have to sell it, and selling takes months, in whatever market you happen to be standing in when you panic. So the error showed up as inventory: thousands of homes bought near the top, now worth less than Zillow paid.
When KeyBanc’s Edward Yruma sampled 650 of them that November - about 20% of what Zillow held - he found two-thirds listed below their purchase price. The company’s own books said it louder: a $304 million write-down in the third quarter. Another $240 to $265 million was already flagged for the fourth, before that quarter had even closed. The loss was still arriving when they announced it.
And the only fast way out of a few thousand illiquid houses was to sell them in bulk to people who buy distressed assets for a living. Zillow lined up roughly seven thousand homes to offload to institutional investors for about $2.8 billion. The sure-fire sign of an irreversible error: a fire-sale.
It Was a Perfect Storm
Rich Barton, the co-founder who by then was running Zillow as chief executive, explained it the other way, and his account deserves a fair hearing.
“Our observed error rate has been far more volatile than we ever expected possible,” he said as the division wound down, and honestly, he was right.
2021 was a genuinely brutal year to forecast house prices, a pandemic market lurching around in ways no model had seen.
If you read this case study charitably, Zillow got unlucky: the once-in-a-generation surge arrived precisely when it had the most money riding on calm.
But notice what that explanation quietly assumes - that the job was to forecast the market correctly, and the failure was forecasting it wrong.
THAT’S NOT THE FAILURE
The market did not kill Zillow Offers. Zillow Offers shouldn’t have been in the market in the first place!
The crucial failure was human. A decision was taken somewhere and backed up in a bunch of other meetings, that this should be something Zillow should do, and do to an extent that their exposure grew to be $500M.
Accuracy is the second question
If you own an AI project right now, this is the part that is actually about you, because the trap that caught Zillow is wearing a friendlier face in your building.
The question in every AI product demo is “how accurate is it?” - and it is the wrong question, or at least the second one. Your model will be wrong sometimes. They all are; the Zestimate still is.
The question that decides whether being wrong is survivable is the one nobody asks in the demo: when it is wrong, what does the wrongness set in motion, and can you take it back?
The solution, is dull, and it works, and it is the opposite of the demo’s instinct.
Keep the model talking - ranking, drafting, recommending to a human who acts - for as long as the action on the far end is expensive and hard to reverse.
Let it act on its own only where a mistake is cheap and you can undo it before lunch.
Wire it straight to the irreversible thing because the demo was impressive, and you have done what Zillow did: removed the one defence you had against the day it is wrong, which is always coming.
That is the difference between a pilot that gets quietly renewed and a half-billion-dollar hole.
Thanks for reading this far! If your name is on a project heading into a review, that question - what does it set in motion, and can you take it back - is one of a handful that decide whether it survives the room. I am putting the checklist I actually use into a short kit, with the script for the renewal meeting itself. If that is you, reply and I will send it over.