Your browser version is outdated. We recommend that you update your browser to the latest version.

Reinforcement learning tutorial

Want to use learning models to look at your data, but don't know where to start? Together with Jill O'Reilly we developed this on-line tutorial, to give beginning modellers a taste of how the use of computational models may help us understand behaviour. We aim to give you a flavour of the mechanics and different steps involved. We focus on learning to implement and fit very basic learning models from two classes of models: one reinforcement learning model, and one normative Bayesian learning model, and show you how you can start to use these models understand your subjects' behaviour. We first developed this tutorial for the Donders Institute summerschool  on Computational Approaches to Decision Making, August 10-14 in Nijmegen, the Netherlands, and by popular demand decided to put it online. We hope that it will help you to take your first steps in the wonderful world of computationals models. 

Computational Psychiatry

A budding and exciting field, if you are interested, then join online for the monthly Trans-continental Computational Psychiatry workgroup!


Model Fitting and Bayesian Stats Toolboxes

If you want to go little more complex and are looking help to navigate the complexities of model fitting, and don't want to write your own code, have a look these excellent toolboxes: 

TAPAS is a collection of algorithms and software tools that are developed by the Translational Neuromodeling Unit at Zurich. These tools have been developed to support translational neuroscience, particularly concerning the application of neuroimaging and computational modeling to research questions in psychiatry and neurology. 

  • Correction of physiological noise in fMRI data.
  • Bayesian inference on computational processes from observed behaviour.
  • Bayesian mixed-effects inference for classification studies.
  • Variational Bayes for linear regression models.
  • Efficient integration of the dynamical systems defined by DCM.

JASP is a great free, open-source statistics toolbox that allows you to do Bayesian (and frequentist) statistics 

Jean Daunizeau wrote a number of excellent toolboxes, for which you can find the code here 

  • Variational Bayesian toolbox for model-based analysis of neurobiological and behavioural data
  • Group-level (random-effect) Bayesian model selection 
  • Reviewing DCM for fMRI  results (SPM8 compatible)
  • Classification / clustering of multivariate data(Gibbs sampling and variational Bayesian algorithms)
  • and... a virtual Rubik's cube!

Dynamic Causal Modelling for fMRI data - getting started

I often get questions about where and how to get started using DCM, so what I will present here is a short summary of what DCM is, and some pointers as to where to go next. 

The aim of dynamic causal modeling (DCM) is to infer the causal architecture of coupled or distributed dynamical systems. It is a Bayesian model comparison procedure that rests on comparing models of how data were generated. DCM was developed for estimating coupling among brain regions and how that coupling is influenced by experimental changes (e.g., time, context or drug). The basic idea is to construct reasonably realistic models of interacting (cortical) regions or nodes. These models are then supplemented with a forward model of how the hidden states of each node (e.g.neuronal activity) map to measured responses (e.g. BOLD resonse). This enables the best model and its parameters (i.e.effective connectivity) to be identified from observed data. Bayesian model comparison is used to select the best model in terms of its evidence (inference on model-space), which can then be characterised in terms of its parameters (inference on parameter-space). DCM allows us to test hypotheses about how nodes communicate; e.g., whether activity in a given neuronal population modulates the coupling between other populations, in a task-specific fashion. (edited from DCM Scholarpedia page)


Where to learn more? 

If you want start using DCM, I would recommend to attend one of the SPM courses, for example


Over the years I have compiled a short (and certainly non-exhaustive) reading list that I always present at the end of my lectures, and thought it would be nice to share with you here. When I have time, I will complete the links below. Until then, pubmed is your friend ;-). 

To get you started 

Some extensions over the years
  • Nonlinear DCM for fMRI: Test whether connectivity between two regions is mediated by another region. (Stephan et al. 2008) 
  • Embedding computational models in DCMs: (den Ouden et al. 2010, J Neurosci.)
  • DCM as a summary statistic: clustering and classification: Classify patients, or even find  new sub-categories (Brodersen et al. 2011, Neuroimage)
  • Integrating tractography and DCM: Prior variance is a good way to embed other forms of information, test validity (Stephan et al. 2009, NeuroImage)
  • Stochastic / spectral DCM: Model resting state studies / background fluctuations (Li et al. 2011 Neuroimage, Daunizeau et al. Physica D 2009)
  • Integrating behavioural and neural data in DCM (2014) Daunizeau et al. PLoS Comput Biol.
And if you like your validation studies, here are some for you
  • reliability (reproducibilty) of parameter estimates (Schuyler et al. 2010) and model selection (Rowe et al. 2010)
  • face validity: simulations and empirical studies (Stephan et al. 2007, 2008)
  • construct validity: comparison with structural equation modelling (Penny et al. 2004)
  • predictive validity
           - infer correct site of seizure origin (David et al. 2008)
           - infer primary recipient of vagal nerve stimulation (Reyt et al. 2010)
           - predict sensory stimulation (Brodersen et al. 2010)
           - predict presence/absence of remote lesion (Brodersen et al. 2011)