Models of Learning

• Introduction

  1. Task description

• Reinforcement Learning

  1. Theory
     • Rescorla-Wagner
     • SoftMax
  2. Data plotting
  3. Simulation
     • Example participants
     • More participants
     • Explore Parameters
     • Grid Search
     • Marginal distributions
     • Summary values
  4. Real data
     • Fit the data
     • Statistical inference
  5. Review
     • Modelling how-to

• Bayesian Learning

  1. Coin tossing
     • Probability
     • Likelihood
     • Uncertainty
     • Learning
     • Review
  2. Bandit task
     • Updating
     • Transition function
     • Learning rate

Developed by
Jill O'Reilly
Hanneke den Ouden
-August 2015

Changing parameter values

Now we can get an intuitive understanding of the learning rate parameter α and the softmax parameter β by changing their values, and seeing how the simulated subject behaves.

Just set the parameters to a different value...

if simulate
     simpars.alpha = % whatever you like between 0 and 1!;
     simpars.beta = % whatever you like as long as it's positive ;
end

...save, and run.

Try this for a range of different parameter values.

• What can you conclude about the effects of α and β?

?

• α
  • If you have a fast learning rate, you can adapt very quickly after a reversal,
    but in the middle of a block, you change the value too readily when the 'unexpected' side is rewarded (you can see this clearly on the single-subject plots)
  • If you have a slow learning rate, your values change very slowly but you are a lot more stable in the middle of the block.
    This is better for the 'low volatility' than for the 'high volatility' environment.
• β
  • When beta is large, the choice probability strongly follows the value difference
  • When beta is small, the choice probability stays around 0.5 regardless of the value difference

  This also means that the choices themselves are more random and so you learn a lot more about the not-so-good option.

OPTIONAL: INITIAL VALUES

We have looked at the effect of the parameters, but one of the assumptions that we haven't talked about yet, is the effect of the initial value that the stimuli have:

• On the very first trial, what do we expect to get when we pick a particular stimulus?

So far, we assumed that the subject doesn't have an initial preference, so the initial value is 0.5 for each stimulus.

But, perhaps you are Dutch and have a particular faith in orange.

Or your next door neighbour told you that the blue slotmachine is really good.

In that case, you might start out believing that one stimulus is better than the other. If you would like to play with this option, then look in the file RLtutorial_simulate.m, change the values of v0:

v0 = .5*ones(1,2); % initial value

You can even set different initial values for each stimulus, for example:

v0 = [0.8 0.5]; % initial value

Then save, and run the RLtutorial_main.m again. What do you conclude about the effect of the initial values?

Don't forget to change the values back to 0.5 when you are done!

►►►