A few days back, commenter igor eduardo kupfer compiled the log5 predictions for the first round, and tried to come up with a test of their validity. We didn't agree on anything, but for the sake of intellectual honesty, here's a breakdown of how those predictions fared, binned in 10% groups (so 0.5-0.6 collects those teams for which the winning probability was between 50-60%):
0.5-0.6: 2-2
0.6-0.7: 3-1
0.7-0.8: 4-1
0.8-0.9: 8-2
0.9-1.0: 9-0
(These records are approximate-- it's possible that I've misremembered a game here or there, but I've just come in from shovelling a foot of snow out of the driveway, and can't be bothered to check.)
So, well, it looks about like you'd expect. The coin-toss games were a coin toss, and the slam-dunk games went as expected. Interestingly, the predictions were wrong about the few predicted upsets by seed (giving Arkansas a 55% chance to take down USC, and Georgia Tech a 70% chance of beating Duke), and the two real upsets that occurred (VCU over Duke, and Winthrop over Notre Dame) were given win probabilities over 80%.
What does this mean? Hell if I know. I'm just reporting.
- Log in to post comments
I should not that I have no idea how those win probabilities were calculated. Following a chain of links from the original Pomeroy article leads to this page giving a formula, but the formula is based only on winning percentages. Which means there's no possible way for 24-7 Notre Dame to be favored 80% of the time against 28-4 Winthrop using that formula, so something else must be going on with these numbers.
I'd be happy to update the results for future rounds, if I could figure out how the hell they were calculated for the first round.
I have no idea how those were calculated either. The way I do it is to use calculate a win-ability estimate, plug that into the log5, and run a simulation for the final probability estimates. It's the win-ability estimate that's the complicated part -- for pro sports, final league win records is usually good enough (except NFL football, where the season isn't long enough to get a good feel for team strength for wins/losses). For NCAA basketball, win/loss records are almost meaningless, since the teams don't play balanced schedules. Some kind of opponent strength correction has to be made -- I have no idea how that was done.
Although now I see that Pomeroy has adjusted offensive/defensive ratings on his stats pages, which he adjusts for competition somehow. With those two numbers you can transform them into a Pythagorean win ability estimator easily enough -- and which he has done (as PYTHAG WINNING PCT on the main stats page). I'm going to guess it was the Pyth number, not win/loss records, that were plugged into the log5, and were used for the simulation.
I do this every season for NBA playoffs, and it's a lot of work, with little reward. One playoffs' worth of results isn't enough to validate the estimates (although it's enough to invalidate them), so skeptics won't be persuaded.