R at Bet Better › R for Betting-Market Data

Chapter 4. Are the odds calibrated?

20 October 2026. Edward Glush. Code run in R 4.4.1 against the live open data on that day.

A price that says 60 per cent should win about 60 times in 100. If it does, the price is calibrated. This chapter tests that claim on tens of thousands of settled outcomes, using the band table published with the open market calibration study.

4.1 The band table

The study took every settled two-way market, removed the margin (chapter 2), and put each side into a band by its fair probability. For each band it reports how many outcomes there were, the average fair probability, and how often the outcome actually happened. We type the published table into a data frame.

bands <- data.frame(
  band     = c("10-20", "20-30", "30-40", "40-50", "50-60", "60-70", "70-80", "80-90"),
  n        = c(2144, 5263, 4909, 5515, 6077, 4945, 5251, 2157),
  implied  = c(17.5, 25.2, 34.7, 45.7, 53.8, 65.2, 74.7, 82.5) / 100,
  observed = c(18.8, 26.7, 37.0, 46.4, 53.2, 63.0, 73.3, 80.9) / 100
)
sum(bands$n)

## [1] 36261

The bands hold 36,261 outcomes. The full study has 36,476. The few under 10 per cent and over 90 per cent are not in the published table.

4.2 The gap in each band

bands$gap_pts <- round(100 * (bands$observed - bands$implied), 1)
bands

##    band    n implied observed gap_pts
## 1 10-20 2144   0.175    0.188     1.3
## 2 20-30 5263   0.252    0.267     1.5
## 3 30-40 4909   0.347    0.370     2.3
## 4 40-50 5515   0.457    0.464     0.7
## 5 50-60 6077   0.538    0.532    -0.6
## 6 60-70 4945   0.652    0.630    -2.2
## 7 70-80 5251   0.747    0.733    -1.4
## 8 80-90 2157   0.825    0.809    -1.6

Every band lands within about two and a half points of the price. That is close. But the signs are not random. Below 50 per cent, outcomes happened a little more often than priced. Above 50 per cent, a little less often. Even after the margin is removed, the long shots were slightly underpriced and the favourites slightly overpriced. Note that this is after proportional removal. Chapter 2 showed that other methods move probability from long shots to favourites, which would change these gaps.

(The gaps can differ from the study page by 0.1 because the page works them out from unrounded figures.)

4.3 Is the gap more than chance?

With a few thousand outcomes per band, some gap is expected by luck alone. A binomial confidence interval tells us how far luck can push the observed rate. binom.test() gives an exact 95 per cent interval from a count of wins and a count of tries. We rebuild the win count from the rate.

bands$wins <- round(bands$n * bands$observed)
ci <- t(mapply(function(x, n) binom.test(x, n)$conf.int, bands$wins, bands$n))
bands$lo <- round(100 * ci[, 1], 1)
bands$hi <- round(100 * ci[, 2], 1)
bands$inside <- with(bands, 100 * implied >= lo & 100 * implied <= hi)
bands[, c("band", "n", "implied", "observed", "lo", "hi", "inside")]

##    band    n implied observed   lo   hi inside
## 1 10-20 2144   0.175    0.188 17.2 20.5   TRUE
## 2 20-30 5263   0.252    0.267 25.5 27.9  FALSE
## 3 30-40 4909   0.347    0.370 35.6 38.4  FALSE
## 4 40-50 5515   0.457    0.464 45.1 47.7   TRUE
## 5 50-60 6077   0.538    0.532 51.9 54.5   TRUE
## 6 60-70 4945   0.652    0.630 61.6 64.3  FALSE
## 7 70-80 5251   0.747    0.733 72.1 74.5  FALSE
## 8 80-90 2157   0.825    0.809 79.2 82.5   TRUE

In four bands the price sits inside the interval: chance alone could explain the gap. In the other four it does not. A z-score says the same thing as one number per band: the gap divided by its standard error. Values beyond about 2 either side are unlikely to be luck.

se <- sqrt(bands$implied * (1 - bands$implied) / bands$n)
bands$z <- round((bands$observed - bands$implied) / se, 2)
bands[, c("band", "gap_pts", "z")]

##    band gap_pts     z
## 1 10-20     1.3  1.58
## 2 20-30     1.5  2.51
## 3 30-40     2.3  3.39
## 4 40-50     0.7  1.04
## 5 50-60    -0.6 -0.94
## 6 60-70    -2.2 -3.25
## 7 70-80    -1.4 -2.33
## 8 80-90    -1.6 -1.96

The two bands either side of the middle, 30-40 and 60-70, show the largest and clearest gaps. A word of caution: we are testing eight bands at once, so one or two large z-scores could appear by chance. Here the signs line up in a clear pattern, which is harder to explain by luck.

4.4 The Brier score

The Brier score is the average squared distance between a forecast and what happened, where what happened is 1 or 0. Lower is better. A forecast of 50 per cent on everything scores 0.25. With only the band table we cannot score each outcome on its own. We can get a close estimate by treating every outcome in a band as if it had the band's average price.

N <- sum(bands$n)
brier_market <- with(bands, sum(n * (observed * (1 - implied)^2 + (1 - observed) * implied^2)) / N)
brier_coin   <- with(bands, sum(n * (observed * 0.25 + (1 - observed) * 0.25)) / N)
brier_ideal  <- with(bands, sum(n * observed * (1 - observed)) / N)
round(c(market = brier_market, coin = brier_coin, perfect_calibration = brier_ideal), 4)

##              market                coin perfect_calibration 
##              0.2181              0.2500              0.2179 

The prices score 0.2181. A coin flip scores 0.25. The third number is what the prices would score if every band were spot on, with no change to how well they sort winners from losers. The difference between the first and third numbers is the cost of the calibration gaps:

reliability <- with(bands, sum(n * (implied - observed)^2) / N)
round(reliability, 5)

## [1] 0.00024

It is tiny: 0.00024. The prices are well calibrated. The gaps are real but small next to the noise in any single game. Most of the Brier score comes from the plain fact that sport is hard to predict, not from bad prices.

4.5 The calibration plot

A calibration plot puts the price on the x axis and what happened on the y axis. A calibrated set of prices sits on the diagonal. Add the confidence intervals as bars.

plot(100 * bands$implied, 100 * bands$observed, pch = 19,
     xlim = c(0, 100), ylim = c(0, 100),
     xlab = "Price says (%)", ylab = "Actually happened (%)",
     main = "Calibration of vig-free prices")
abline(0, 1, lty = 2)
arrows(100 * bands$implied, bands$lo, 100 * bands$implied, bands$hi,
       angle = 90, code = 3, length = 0.03)

At this scale the points sit almost on the line. Zoom in on the y axis (for example ylim = c(10, 90) and plot the gap instead of the rate) to see the tilt: points above zero on the left, below zero on the right.

4.6 With the outcome-level file

If you download the outcome-level CSV from the Kaggle mirror, you can do all of this from raw rows and get an exact Brier score. The columns are market_implied_probability (margin removed) and outcome (1 or 0). The code is short:

cal <- read.csv("market_calibration.csv")
mean((cal$market_implied_probability - cal$outcome)^2)   # exact Brier score
cal$bin <- cut(cal$market_implied_probability, seq(0, 1, 0.1), include.lowest = TRUE)
aggregate(cbind(market_implied_probability, outcome) ~ bin, data = cal, FUN = mean)

That block is not run here, because the file sits behind a Kaggle sign-in. Everything above it was run on the published table.

Try it yourself

• Fit a straight line of observed on implied, weighted by n. A slope below 1 is the long-shot tilt in one number.
• Swap binom.test() for the normal approximation and compare the intervals.
• Work out how many outcomes a band would need before a 0.7 point gap became clear of chance.

Data licensed CC BY 4.0, free to reuse with a credit. Model estimates, not advice. 18+. If gambling is causing you harm: gamblinghelponline.org.au.