The effectiveness of learning analytics for identifying at-risk students in higher education – an extra titbit

I’ve recently published a paper with a colleague Dr Rebecca Siddle for a special edition of Assessment and Evaluation in Higher Education, The paper explored the impact of ‘no-engagement’ alerts for identifying students in need of further support and contrasted it with using background characteristics. In this paper, we used widening participation status as a proxy for poorer socio-economic background. The essence of the paper is that:

  • Learning analytics is an effective way to target learners in need of further support. The ‘no-engagement’ alerts were effective at providing early warnings. As students generated more alerts they were both less likely to progress to the next year and less likely to achieve higher grades.
  • We don’t use socio-economic factors in our core algorithm, nonetheless, students from disadvantaged backgrounds were still more likely to be identified because they engaged less with academic activities.
  • The ‘no-engagement’ alerts are more efficient at identifying students at risk than background characteristics. If we base alerts on WP status, we will generate lots of false positive alerts (about three for every student we successfully identify as being at risk). However, if we use ‘no-engagement’ alerts we generate less than one false positive alert for every accurate alert raised.

We took a strong view when we first started using learning analytics that demographics shouldn’t be in the algorithm for deciding risk. I still feel that’s the right approach, the role of background in algorithms is rightly a cause for concern. We didn’t include the following graph in the paper (for lack of space), but I think it helps make an important point. On average, Widening Participation students had a lower average engagement rating than their peers. But if you look at the two bars, WP students engagement is bifurcated, pushed to the extremes, they are both more likely to be highly-engaged and lowly-engaged than their peers (*1). Targeting students using background would target both sections of the WP cohort.

fullsizeoutput_6

There’s room for considering strategies for targeting within cohorts of historically disadvantaged groups, but we remain concerned about adding background into algorithms.

Ed Foster & Rebecca Siddle (2019) The effectiveness of learning analytics for identifying at-risk students in higher education, Assessment & Evaluation in Higher Education, DOI: 10.1080/02602938.2019.1682118

(*1) We see the same pattern in most groups with poorer attainment. The one exception is male students who are just less engaged on average than their female peers.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.