Show HN is where builders launch in front of the internet's harshest crowd. About 98% of posts flop. About 2% reach the front page and get seen by millions.
I wanted to know how much of that outcome is already visible in the title.
Setup
- 5,200
- real Show HN posts
- $0.16
- total cost
- 5.8 min
- wall time
- 0
- failed requests
Jev saw only the title of each post and gave a probability that it would reach the front page. No cherry-picking.
It ranks better than chance
The ranking has an AUC of 0.686. Posts it scored higher really were more likely to make the front page.
But the confidence is off
Ranking is one question. Calibration is another: when Jev says "30% chance", does it happen 30% of the time?
It does not. The calibration curve sits well below the diagonal. Jev is overconfident about how likely success is.
The twist
I also asked two "obvious" proxy questions. Both scored worse than a coin flip.
- 0.686
- direct front-page probability
- 0.383
- “does this solve a real problem?”
- 0.422
- “does the title say what it does?”
The intuitive proxies point backwards. Only the direct probability estimate carries real signal.
The biggest miss
A post titled "Elevators". Jev gave it 22%. It got 1,680 points.
What I take from it
Ranking and calibration are different skills, and you can have one without the other. A model that orders things correctly is useful for sorting a queue. A model you plan to trust on "I'm 90% sure" needs its confidence checked against real outcomes before anything acts on it.
The full data and code are in the thread.