The Technion Prediction Tournament
Organized by: Ido Erev, Eyal Ert, and Alvin E. Roth
Posted on September 2, 2008
Fourteen teams participated in the three competitions. The typical team participated in two competitions, and the average number of submissions per competition was 8. The teams used a large span of methods ranging from logistic regression, ACT-R based cognitive modeling, neural networks, production rules, and basic mathematical models.
In accordance with the competition rules, the ranking of the models was determined based on the mean squared distance (MSD) between the predicted and observed choice proportion in the competition data set. In addition to this statistic we present the proportion of agreement (Pagree) and the correlation between the predicted and observed proportions, and the model’s ENO (equivalent number of observations). ENO is a rank preserving transformation of the MSD score that estimates the value of the models in terms of the expected size of experiment that has to be run to provide predictions that are as accurate as the model.
Condition Description:
Table 1 presents the three best submitted-models, and the best baseline model for condition Description. The winner of this competition is the model submitted by Ernan Haruvy. This model uses a logistic regression format. Interestingly, however, the best baseline model (CPT with cumulative normalization) provides slightly better predictions.
Table 1: Main results for condition Description.
|
|
Fitness scores based on the estimation set (S2=.1860) |
Prediction scores based on the competition set (S2=.1636) |
||||||
|
Title |
Team and idea |
Pagree |
Corr |
MSD |
Pagree |
Corr |
MSD |
ENO |
|
Winner |
Haruvy: logistic regression |
88% |
0.92 |
0.0099 |
90% |
0.94 |
0.0126 |
56.36 |
|
Runner up |
Yechiam: Case-based expectancies |
92% |
0.89 |
0.0141 |
91% |
0.93 |
0.0133 |
31.95 |
|
Second runner up |
Ahn & Picard: CPT with Aspiration levels |
87% |
0.93 |
0.0088 |
90% |
0.92 |
0.0165 |
19.66 |
|
Best baseline |
SCPT with normalization |
89% |
0.92 |
0.0116 |
95% |
0.95 |
0.0102 |
80.99 |
Condition E-Sampling
Table 2 presents the three best submitted-models, and the best baseline model for condition E-Sampling. The winner in this competition is the model submitted by Stefan Herzog, Robin Hau, and Ralph Hertwig. This model assumes a linear combination of four rules: primed sampling, Cumulative Prospect Theory, Priority heuristic and natural-mean heuristic (see Hertwig & Pleskac, 2008).
Table 2: Main results for Condition E-Sampling
|
|
Fitness scores based on the estimation set (S2=.2023) |
Prediction scores based on the competition set (S2 =.2111) |
||||||
|
Title |
Team and idea |
Pagree |
Corr |
MSD |
Pagree |
Corr |
MSD |
ENO |
|
Winner |
Herzog, Hau, Hertwig. Linear Combination |
95% |
0.92 |
0.0099 |
83% |
0.8 |
0.0187 |
25.92
|
|
Runner up |
Ahn, Picard: Sample by CPT and aspiration levels |
92% |
0.9 |
0.0115 |
82% |
0.82 |
0.0203 |
21.662 |
|
Second runner up |
Hau, Hertwig Weighted samples |
95% |
0.89 |
0.01548 |
82% |
0.79 |
0.025 |
14.614
|
|
Best baseline |
Primed sampler with variability |
95% |
0.88 |
0.017 |
82% |
0.8 |
0.0244 |
15.23 |
Condition E -Repeated
Table 3 presents the three best submitted-models, and the best baseline model for condition E-Repeated. The winner in this competition is the model submitted by Terry Stewart, Robert West, and Christian Liebre. This model uses ACT-R architecture and assumes reliance similarity based reasoning. Implicit in this abstraction is the assumption of high sensitivity of small set of previous experiences in situations that are perceived to be similar to the current choice task. The best baseline model (explorative sampler with recency) that can be described as a different abstraction of the idea of reliance on small set of experiences provides slightly better predictions.
Table 3: Main results for Condition E -Repeated
|
|
Prediction scores based on the estimation set (S2 = .0875) |
Fitness scores based on the competition set (S2 = .0928) |
||||||
|
Title |
Team and idea |
Pagree |
Corr |
MSD |
Pagree |
Corr |
MSD |
ENO |
|
Winner |
Stewart, West, & Lebiere: ACT – R with sequential dependencies and blending memory |
77% |
0.88 |
0.0094 |
87% |
0.89 |
0.0075 |
32.50 |
|
Runner up |
Hochman & Ayal: Two- stage sampler |
80% |
0.9 |
0.00647 |
83% |
0.87 |
0.0084 |
24.71 |
|
Second runner up |
Haruvy: Normalized Reinforcement Learning with inertia |
75% |
0.86 |
0.008 |
86% |
0.85 |
0.0084 |
24.71
|
|
Best baseline |
Explorative sampler with recency |
82% |
0.88 |
0.0075 |
86% |
0.89 |
0.0066 |
47.22 |