An IRT forecasting model: linking proper scoring rules to item response theory
Journal Title: Judgment and Decision Making - Year 2017, Vol 12, Issue 2
Abstract
This article proposes an Item Response Theoretical (IRT) forecasting model that incorporates proper scoring rules and provides evaluations of forecasters’ expertise in relation to the features of the specific questions they answer. We illustrate the model using geopolitical forecasts obtained by the Good Judgment Project (GJP) (see Mellers, Ungar, Baron, Ramos, Gurcay, Fincher, Scott, Moore, Atanasov, Swift, Murray, Stone & Tetlock, 2014). The expertise estimates from the IRT model, which take into account variation in the difficulty and discrimination power of the events, capture the underlying construct being measured and are highly correlated with the forecasters’ Brier scores. Furthermore, our expertise estimates based on the first three years of the GJP data are better predictors of both the forecasters’ fourth year Brier scores and their activity level than the overall Brier scores obtained and Merkle’s (2016) predictions, based on the same period. Lastly, we discuss the benefits of using event-characteristic information in forecasting.
Authors and Affiliations
Yuanchao Emily Bo, David V. Budescu, Charles Lewis, Philip E. Tetlock and Barbara Mellers
Keywords
Related Articles
- EP ID EP678279
- DOI -
- Views 160
- Downloads 0
Modeling sequential context effects in judgment analysis:
In this article a broad perspective incorporating elements of time series theory is presented for conceptualizing the data obtained in multi-trial judgment experiments. Recent evidence suggests that sequential context ef...
Sense of control matters: a long spatial distance leads to a short-term investment preference
Increasingly, we can invest in projects that are distributed around the world through online investment platforms. Will the spatial distance between these projects and ourselves affect our investment preferences? The pre...
Pace yourself: Improving time-saving judgments when increasing activity speed
The time-saving bias describes people’s tendency to misestimate the time they can save by increasing the speed in which they perform an activity such as driving or completing a task. People typically underestimate time s...
Energy conservation goals: What people adopt, what they recommend, and why
Failures to reduce greenhouse gas emissions by adopting policies, technologies, and lifestyle changes have led the world to the brink of crisis, or likely beyond. Here we use Internet surveys to attempt to understand the...
The Regret Elements Scale: Distinguishing the affective and cognitive components of regret
Regret is one of the most common emotions, but researchers generally measure it in an ad-hoc, unvalidated fashion. Three studies outline the construction and validation of the Regret Elements Scale (RES), which distingui...