The Technion Prediction Tournament
Organized by: Ido Erev, Eyal Ert, and Alvin E. Roth
7.3.1 Normalized Reinforcement Learning (NRL)
Download SAS example of the NRL model here
The normalized reinforcement learning model (see Erev & Barron, 2005 and a similar model in Erev, Bereby-Meyer & Roth, 1999) assumes a stochastic choice rule that is similar to the SCPT rule. Specifically, the probability of selecting the risky prospect at trial t is given by:
where WVt(k) is the weighted value of action k at trial t, μ is a free payoff sensitivity parameter, and Dt is a measure of experienced payoff variability. If strategy k was selected at t, its weighted value at trial t+1 is a weighted average of WVt(k) at t, and vt the obtained payoff at t
The parameter 0 < ω < 1 captures the weight for the recent outcomes. The initial value, WV1(k) is assumed to equal A(1)—the expected payoff from random choice (e.g., A(1) in Problem 1 is .5[18.8(.8) + 7.6(.2)] +15.5].
The payoff variability term Dt is the weighted average of the difference between the obtained payoff at trial t and t-1:
where v0 is assumed to equal A(1), and D1 is assumed to equal l.
The predictions of this model with the parameters estimated by Erev and Barron and the parameters that best fit the data are presented in Table 4. The results reveal a relatively large advantage of the estimated parameters.
7.3.2 Basic Reinforcement Learning
The basic reinforcement learning model, considered here, is a simplification of the NRL model. The simplification involves the assumption Dt =1. Table 4 reveals that this simplification impairs the fit.
7.3.3 RELACS
Erev and Barron (2005) present a generalization of the NRL model that assumes reinforcement learning among cognitive strategies. Since this model is rather complex, we do not present it here. Table 4 presents its predictions for the current task with the parameters estimated by Erev and Barron, and with the parameters that best fit the current data. The results reveal that RELACS provide relatively good predictions using the original parameter, but the best fit is not very good.
7.3.4 Explorative sampler
The explorative sample model (Erev, Ert & Yechiam, 2008) can be summarized with the following assumptions:
A1: Exploration and exploitation. The agents are assumed to consider two cognitive strategies: exploration and exploitation.
Exploration implies a random choice. The probability of exploration is 1 in the very first trial, and (when information concerning the forgone payoffs is not available)it reduces toward an asymptote (at ε) with experience. The effect of experience on the probability of exploration depends on the expected length of the experiment (T). Exploration diminishes quickly when T is small, and slowly when T is large. This assumption is quantified as follows:
where d is a free parameter that captures the sensitivity to the length of the experiment.
A2: Experiences. The experiences with each alternative include the set of observed outcomes yielded by this alternative in previous trials. In addition, when the payoffs are limited to the obtained payoff, the subjective value of the very first outcome is recalled as an experience with all the alternatives.
A3: Naïve sampling. Under exploitation the agent draws (with replacement) a sample of mt past experiences with each alternative. All previous experiences are equally likely to be sampled.
A4: Sampling algorithm. The value of mt at trial t is assumed to be randomly selected from the set {1, 2,…… k} where k is a free parameter.
The sampling algorithm is assumed to depend on the available information. When the feedback is limited to the obtained payoffs the sampling from the experiences with the different alternatives is independent. When the foregone payoffs are known (the decision makers receive complete feedback that includes the payoff from the unselected alternatives), the distinct samples are perfectly correlated. The decision maker selects one set of mt trials, and the outcomes in those trials are used to determine the values of the different alternatives.
A5: Regressiveness, diminishing sensitivity, and choice. The recalled subjective values of the outcome x (from selecting alternative j) at trial t is assumed to be affected by two factors: regression to the mean of all the experiences with the relevant alternative (in the first t-1 trials), and diminishing sensitivity. Regression is captured with the assumption that the regressed value is Rx= (1-w)x + (w)Aj(t), where w is a free parameter and Aj(t) is the average outcome from the relevant alternative.[1]
Diminishing sensitivity is captured with a variant of prospect theory’s (Kahneman and Tversky, 1979) value function that assumes
Where αt = (1+Vt)(-β), β > 0 is a free parameter, and Vt is a measure of payoff variability. Vt is computed as the average absolute difference between consecutive obtained payoffs in the first t-1 trials (with an initial value at 0). The parameter β captures the effect of diminishing sensitivity: large β implies quick increase in diminishing sensitivity with payoff variability.
The estimated subjective value of each alternative at trial t is the mean of the subjective value of the alternative's sample in that trial. Under exploitation the agent selects the alternative with the highest estimated value.
Table 4 presents the predictions of the explorative sampler model with the parameters estimated by Erev et al. (2008), and with the parameters that best fit the current data. The results reveal that the model tends to over-predict the tendency to select the risky prospect.
7.3.5 Explorative sampler with recency
Download SAS example of the Explorative sampler with recency model here
The last model presented here is a refinement of the explorative sampler model that was developed to capture the bias considered above. Specifically, the refined model assumes that the most recent outcome with each alternative is always considered. This change is implemented by replacing assumption A3 with the following assumption:
A3’: Naïve sampling with recency. Under exploitation the agent draws (with replacement) a sample of mt past experiences with each alternative. The first draw is the most recent experience with each alternative. All previous experiences are equally likely to be sampled in the remaining mt-1 draws.
Notice that Assumption A3’ implies a hot stove effect (see Denrell & March, 2001): An increase in risk aversion with experience.
The right hand column in table 4 presents the predictions of the refined model. The results show that the refinement improves the fit. Additional analysis shows that the added recency effect does not impair the prediction of the explorative sampling models in the experimental condition reviewed by Erev & Haruvy, 2007.
Table 4: Condition E-repeated: The upper panel presents the aggregated proportion of choices in Risk (Prisk) and the predictions of the baseline models. The lower panel presents summary statistic.
|
Repeated Decisions from Experience (E- Repeated) |
||||||||||||||
|
|
Risk |
Safe |
observed |
Normalized Reinforcement Learning (NRL) |
Basic Reinforcement Learning (RL) |
RL among cog. Strategies (RELACS) |
Explorative sampler
(δ = 0.55, and w=.3 in all cases) |
|||||||
|
# |
High |
Phigh |
Low |
Med |
Prisk |
Org w=.01 λ=3 |
Fit w=.15 λ=1.1 |
Org w=.03 λ=1 |
Fit W=.15 λ=1 |
Org l = 8 a=.0012 b=0.2 k=4 |
Fit l = 8 a=.005 b=0.1 k=4 |
Org. β=.15 ε=.08 k = 5 |
Fit β=.05 ε=.12 k=20 |
With recency β=.10 ε=.12 k=8 |
|
1 |
-0.3 |
0.96 |
-2.1 |
-0.3 |
0.33 |
0.43 |
0.31 |
0.49 |
0.48 |
0.34 |
0.31 |
0.25 |
0.27 |
0.27 |
|
2 |
-0.9 |
0.95 |
-4.2 |
-1.0 |
0.50 |
0.49 |
0.41 |
0.49 |
0.48 |
0.62 |
0.57 |
0.60 |
0.52 |
0.50 |
|
3 |
-6.3 |
0.30 |
-15.2 |
-12.2 |
0.24 |
0.47 |
0.40 |
0.46 |
0.33 |
0.32 |
0.31 |
0.37 |
0.35 |
0.33 |
|
4 |
-10.0 |
0.20 |
-29.2 |
-25.6 |
0.32 |
0.50 |
0.41 |
0.50 |
0.30 |
0.32 |
0.32 |
0.36 |
0.41 |
0.36 |
|
5 |
-1.7 |
0.90 |
-3.9 |
-1.9 |
0.45 |
0.51 |
0.47 |
0.50 |
0.49 |
0.60 |
0.57 |
0.67 |
0.52 |
0.45 |
|
6 |
-6.3 |
0.99 |
-15.7 |
-6.4 |
0.68 |
0.58 |
0.61 |
0.50 |
0.49 |
0.76 |
0.75 |
0.72 |
0.68 |
0.69 |
|
7 |
-5.6 |
0.70 |
-20.2 |
-11.7 |
0.37 |
0.55 |
0.57 |
0.65 |
0.52 |
0.60 |
0.64 |
0.65 |
0.68 |
0.61 |
|
8 |
-0.7 |
0.10 |
-6.5 |
-6.0 |
0.27 |
0.49 |
0.35 |
0.51 |
0.50 |
0.27 |
0.27 |
0.23 |
0.34 |
0.27 |
|
9 |
-5.7 |
0.95 |
-16.3 |
-6.1 |
0.43 |
0.51 |
0.44 |
0.49 |
0.41 |
0.64 |
0.59 |
0.67 |
0.56 |
0.54 |
|
10 |
-1.5 |
0.92 |
-6.4 |
-1.8 |
0.44 |
0.50 |
0.44 |
0.49 |
0.46 |
0.60 |
0.56 |
0.65 |
0.51 |
0.49 |
|
11 |
-1.2 |
0.02 |
-12.3 |
-12.1 |
0.26 |
0.43 |
0.24 |
0.50 |
0.49 |
0.20 |
0.21 |
0.14 |
0.19 |
0.17 |
|
12 |
-5.4 |
0.94 |
-16.8 |
-6.4 |
0.55 |
0.57 |
0.55 |
0.53 |
0.50 |
0.69 |
0.69 |
0.75 |
0.66 |
0.56 |
|
13 |
-2.0 |
0.05 |
-10.4 |
-9.4 |
0.11 |
0.36 |
0.17 |
0.43 |
0.35 |
0.18 |
0.15 |
0.16 |
0.19 |
0.18 |
|
14 |
-8.8 |
0.60 |
-19.5 |
-15.5 |
0.66 |
0.59 |
0.62 |
0.72 |
0.75 |
0.64 |
0.69 |
0.72 |
0.77 |
0.71 |
|
15 |
-8.9 |
0.08 |
-26.3 |
-25.4 |
0.19 |
0.50 |
0.34 |
0.54 |
0.48 |
0.27 |
0.30 |
0.22 |
0.34 |
0.26 |
|
16 |
-7.1 |
0.07 |
-19.6 |
-18.7 |
0.34 |
0.45 |
0.27 |
0.50 |
0.43 |
0.23 |
0.23 |
0.18 |
0.29 |
0.22 |
|
17 |
-9.7 |
0.10 |
-24.7 |
-23.8 |
0.37 |
0.52 |
0.39 |
0.56 |
0.50 |
0.30 |
0.34 |
0.25 |
0.38 |
0.29 |
|
18 |
-4.0 |
0.20 |
-9.3 |
-8.1 |
0.34 |
0.47 |
0.37 |
0.48 |
0.44 |
0.28 |
0.27 |
0.32 |
0.35 |
0.32 |
|
19 |
-6.5 |
0.90 |
-17.5 |
-8.4 |
0.49 |
0.58 |
0.60 |
0.58 |
0.56 |
0.69 |
0.73 |
0.72 |
0.69 |
0.63 |
|
20 |
-4.3 |
0.60 |
-16.1 |
-4.5 |
0.08 |
0.28 |
0.10 |
0.16 |
0.06 |
0.32 |
0.27 |
0.11 |
0.13 |
0.13 |
|
21 |
2.0 |
0.10 |
-5.7 |
-4.6 |
0.11 |
0.43 |
0.27 |
0.46 |
0.39 |
0.28 |
0.25 |
0.23 |
0.27 |
0.24 |
|
22 |
9.6 |
0.91 |
-6.4 |
8.7 |
0.41 |
0.48 |
0.40 |
0.43 |
0.28 |
0.49 |
0.45 |
0.59 |
0.45 |
0.45 |
|
23 |
7.3 |
0.80 |
-3.6 |
5.6 |
0.39 |
0.48 |
0.42 |
0.44 |
0.29 |
0.38 |
0.35 |
0.52 |
0.40 |
0.35 |
|
24 |
9.2 |
0.05 |
-9.5 |
-7.5 |
0.08 |
0.35 |
0.16 |
0.37 |
0.23 |
0.21 |
0.17 |
0.16 |
0.20 |
0.20 |
|
25 |
7.4 |
0.02 |
-6.6 |
-6.4 |
0.19 |
0.44 |
0.25 |
0.51 |
0.50 |
0.21 |
0.22 |
0.16 |
0.19 |
0.18 |
|
26 |
6.4 |
0.05 |
-5.3 |
-4.9 |
0.20 |
0.47 |
0.30 |
0.52 |
0.50 |
0.26 |
0.28 |
0.17 |
0.27 |
0.22 |
|
27 |
1.6 |
0.93 |
-8.3 |
1.2 |
0.50 |
0.48 |
0.39 |
0.46 |
0.38 |
0.52 |
0.48 |
0.56 |
0.47 |
0.46 |
|
28 |
5.9 |
0.80 |
-0.8 |
4.6 |
0.58 |
0.51 |
0.48 |
0.49 |
0.43 |
0.41 |
0.41 |
0.54 |
0.48 |
0.42 |
|
29 |
7.9 |
0.92 |
-2.3 |
7.0 |
0.51 |
0.53 |
0.50 |
0.50 |
0.45 |
0.56 |
0.54 |
0.70 |
0.57 |
0.50 |
|
30 |
3.0 |
0.91 |
-7.7 |
1.4 |
0.41 |
0.58 |
0.58 |
0.57 |
0.54 |
0.59 |
0.62 |
0.70 |
0.68 |
0.59 |
|
31 |
6.7 |
0.95 |
-1.8 |
6.4 |
0.52 |
0.51 |
0.42 |
0.48 |
0.44 |
0.58 |
0.54 |
0.64 |
0.54 |
0.53 |
|
32 |
6.7 |
0.93 |
-5.0 |
5.6 |
0.49 |
0.56 |
0.54 |
0.52 |
0.47 |
0.60 |
0.59 |
0.73 |
0.63 |
0.52 |
|
33 |
7.3 |
0.96 |
-8.5 |
6.8 |
0.65 |
0.52 |
0.46 |
0.48 |
0.38 |
0.61 |
0.57 |
0.65 |
0.58 |
0.56 |
|
34 |
1.3 |
0.05 |
-4.3 |
-4.1 |
0.30 |
0.49 |
0.34 |
0.51 |
0.51 |
0.26 |
0.27 |
0.18 |
0.27 |
0.21 |
|
35 |
3.0 |
0.93 |
-7.2 |
2.2 |
0.44 |
0.54 |
0.50 |
0.50 |
0.44 |
0.58 |
0.57 |
0.72 |
0.59 |
0.52 |
|
36 |
5.0 |
0.08 |
-9.1 |
-7.9 |
0.09 |
0.45 |
0.28 |
0.48 |
0.39 |
0.27 |
0.27 |
0.20 |
0.31 |
0.24 |
|
37 |
2.1 |
0.80 |
-8.4 |
1.3 |
0.28 |
0.40 |
0.29 |
0.35 |
0.20 |
0.32 |
0.28 |
0.38 |
0.28 |
0.26 |
|
38 |
6.7 |
0.07 |
-6.2 |
-5.1 |
0.29 |
0.43 |
0.25 |
0.47 |
0.39 |
0.26 |
0.24 |
0.18 |
0.27 |
0.23 |
|
39 |
7.4 |
0.30 |
-8.2 |
-6.9 |
0.58 |
0.61 |
0.63 |
0.77 |
0.80 |
0.68 |
0.74 |
0.65 |
0.74 |
0.67 |
|
40 |
6.0 |
0.98 |
-1.3 |
5.9 |
0.61 |
0.53 |
0.50 |
0.50 |
0.48 |
0.67 |
0.62 |
0.62 |
0.61 |
0.58 |
|
41 |
18.8 |
0.80 |
7.6 |
15.5 |
0.52 |
0.56 |
0.57 |
0.60 |
0.55 |
0.64 |
0.67 |
0.65 |
0.67 |
0.59 |
|
42 |
17.9 |
0.92 |
7.2 |
17.1 |
0.48 |
0.53 |
0.47 |
0.49 |
0.41 |
0.62 |
0.59 |
0.67 |
0.54 |
0.49 |
|
43 |
22.9 |
0.06 |
9.6 |
9.2 |
0.88 |
0.65 |
0.71 |
0.63 |
0.70 |
0.86 |
0.90 |
0.90 |
0.87 |
0.87 |
|
44 |
10.0 |
0.96 |
1.7 |
9.9 |
0.56 |
0.47 |
0.37 |
0.47 |
0.42 |
0.61 |
0.54 |
0.38 |
0.39 |
0.38 |
|
45 |
2.8 |
0.80 |
1.0 |
2.2 |
0.48 |
0.57 |
0.60 |
0.53 |
0.55 |
0.67 |
0.72 |
0.70 |
0.72 |
0.65 |
|
46 |
17.1 |
0.10 |
6.9 |
8.0 |
0.32 |
0.46 |
0.30 |
0.49 |
0.42 |
0.25 |
0.25 |
0.23 |
0.30 |
0.27 |
|
47 |
24.3 |
0.04 |
9.7 |
10.6 |
0.25 |
0.39 |
0.19 |
0.46 |
0.38 |
0.19 |
0.18 |
0.15 |
0.21 |
0.18 |
|
48 |
18.2 |
0.98 |
6.9 |
18.1 |
0.59 |
0.53 |
0.48 |
0.48 |
0.44 |
0.69 |
0.62 |
0.55 |
0.51 |
0.52 |
|
49 |
13.4 |
0.50 |
3.8 |
9.9 |
0.13 |
0.44 |
0.35 |
0.35 |
0.19 |
0.35 |
0.32 |
0.31 |
0.26 |
0.25 |
|
50 |
5.8 |
0.04 |
2.7 |
2.8 |
0.35 |
0.48 |
0.34 |
0.50 |
0.50 |
0.23 |
0.23 |
0.16 |
0.23 |
0.19 |
|
51 |
13.1 |
0.94 |
3.8 |
12.8 |
0.52 |
0.48 |
0.39 |
0.46 |
0.39 |
0.61 |
0.54 |
0.54 |
0.48 |
0.44 |
|
52 |
3.5 |
0.09 |
0.1 |
0.5 |
0.26 |
0.45 |
0.30 |
0.49 |
0.47 |
0.23 |
0.21 |
0.21 |
0.27 |
0.25 |
|
53 |
25.7 |
0.10 |
8.1 |
11.5 |
0.11 |
0.38 |
0.19 |
0.32 |
0.15 |
0.20 |
0.17 |
0.20 |
0.22 |
0.20 |
|
54 |
16.5 |
0.01 |
6.9 |
7.0 |
0.18 |
0.42 |
0.26 |
0.50 |
0.49 |
0.18 |
0.18 |
0.12 |
0.17 |
0.16 |
|
55 |
11.4 |
0.97 |
1.9 |
11.0 |
0.66 |
0.58 |
0.53 |
0.51 |
0.49 |
0.72 |
0.70 |
0.77 |
0.66 |
0.65 |
|
56 |
26.5 |
0.94 |
8.3 |
25.2 |
0.53 |
0.55 |
0.51 |
0.51 |
0.39 |
0.66 |
0.66 |
0.73 |
0.62 |
0.55 |
|
57 |
11.5 |
0.60 |
3.7 |
7.9 |
0.45 |
0.53 |
0.52 |
0.55 |
0.48 |
0.52 |
0.53 |
0.52 |
0.56 |
0.48 |
|
58 |
20.8 |
0.99 |
8.9 |
20.7 |
0.63 |
0.58 |
0.61 |
0.50 |
0.48 |
0.74 |
0.73 |
0.68 |
0.67 |
0.67 |
|
59 |
10.1 |
0.30 |
4.2 |
6.0 |
0.32 |
0.49 |
0.43 |
0.49 |
0.43 |
0.34 |
0.33 |
0.38 |
0.40 |
0.37 |
|
60 |
8.0 |
0.92 |
0.8 |
7.7 |
0.44 |
0.46 |
0.38 |
0.47 |
0.40 |
0.57 |
0.51 |
0.53 |
0.43 |
0.43 |
|
Mean |
0.40 |
0.49 |
0.41 |
0.49 |
0.44 |
0.46 |
0.46 |
0.45 |
0.41 |
0.40 |
||||
|
MSD*100 |
2.36 |
0.92 |
3.01 |
2.24 |
1.21 |
1.10 |
1.92 |
1.15 |
0.75 |
|||||
|
Correlation |
0.81 |
0.83 |
0.53 |
0.56 |
0.87 |
0.88 |
0.84 |
0.85 |
0.88 |
|||||
|
Pagree |
0.74 |
0.76 |
0.60 |
0.67 |
0.77 |
0.77 |
0.73 |
0.75 |
0.82 |
|||||
[1] Implicit in this regressiveness (the assumption W > 0) is the assumption that all the experiences are weighted (because all the experiences affect the mean). The value of this implicit assumption was demonstrated by Lebiere, Gonzalez and Martin (2006): It is necessary to capture the observed behavior in Problem 24.