Connecting Learning Analytics with earlier work on Student Retention

Between 2008 and 2011, we led a small research consortium looking at those factors that caused students to consider leaving early and those factors that helped them to stay for the Higher Education Funding Council/ Paul Hamlyn-funded “What Works? Student Retention and Success”(1) programme. The work we carried out for the HERE project profoundly shaped our thinking on learning analytics and I wanted to upload some of our core documents here.

“What Works? Student Retention & Success”

In 2007, the UK National Audit Office produced a report into HE student retention: “Staying the Course”. It identified a number of factors associated with risks of early departure (socio-economic background, entry qualifications etc.) and then identified what different stakeholders across the sector were doing to improve retention. It was generally positive about the efforts that the sector had carried out to support student success. However, the MPs on the Public Accounts Committee (PAC) who received the report “felt that a significant barrier to further progress was the lack of evidence about what actually works to improve student retention and completion.” (Thomas, 2012, P7.)

In response, HEFCE partnered with the Paul Hamlyn Foundation and set up a research programme. The “What Works? Student Retention and Success” project was co-ordinately by the Higher Education Academy. Seven research teams representing 22 different institutions conducted studies into a range of activities including summer schools, induction, peer mentoring, team and problem-based learning and enhanced personal tutoring.

Mainstreaming Engagement and Belonging

The final report (Thomas (2012)) had many recommendations, but two important themes for anyone interested in using learning analytics for student success. I have added a rather lengthy quote below from the report (text in bold my addition).

 

“This study has considered both student retention and success. During the course of this programme we have sharpened our understanding of success. It has become increasingly clear that ‘success’ means helping all students to become more engaged and more effective learners in higher education, … In line with this understanding of success and underpinned by the What Works? findings the study advocates a mainstream approach to improving the retention and success of all students. It can be difficult to know which students are most likely to withdraw on the basis of student entry characteristics alone; mainstream approaches reach all students, particularly those who are considering withdrawing, and enable each student to maximise their success. This can be supplemented by paying attention to the ways in which students integrate, behave and perform once they are in higher education through a range of engagement indicators, and then intervening if necessary. 

The findings of this programme present a compelling case that in higher education, belonging is critical to student retention and success. Although other studies have pointed to this and many staff in universities would readily accept this contention, we argue that the implications are very often not addressed in institutional priorities, policies, processes and practices. Where strategies are employed to boost student engagement, they are often focused on narrow groups of students, and situated outside of the academic domain, thus failing to meet the needs of the much larger number of students that the What Works? programme indicates may be at risk of withdrawal or underachievement.”

What did we do?

The HERE Project

The HERE Project was a partnership piece of work between Nottingham Trent University, the University of Bradford and Bournemouth University. We produced a quite lengthy report and appendices (in a slightly clunky format to suit the funders (sorry)), full of interesting content. We investigated two areas:

• Doubting
• The impact of course teams on success

Doubting

Doubting is a perfectly normal part of life. I’d argue that anyone who didn’t have a few moments of doubt after starting their first year in university would have either have to be supremely confident or incredibly lucky. In this context we defined doubters as “students who have doubts about being on the right course/ right university that are sufficiently strong to have considered withdrawal.” (Foster, et al, 2012)

Essentially we used a range of first year surveys to ask students whether or not they’d considered leaving and a range of other questions about their experience. We followed these up with interviews and focus groups and then (with students’ permission) analysed the relationship between doubting and success.

In our studies doubters reported: 

• A less satisfactory academic experience
• Lower levels of understanding about the differences between FE & HE 
• Lower levels of confidence 
• Being more likely to be working “very hard” or “not very hard at all” (in other words frantically engaged, or scarcely engaged with their studies) 
• That they were more likely to be struggling with their studies and less confident about asking for help
• That they were less likely to be enjoying being a student

Ultimately, more doubters left early than their non-doubting peers. However, on average all respondents to our surveys were more likely to progress than those who didn’t respond (yes this is a problem with all such research methodologies). Importantly thought, this chimed with earlier work by Ozga & Sukhnandan (1998) (2), departure is often a process with potential early warning signs rather than a sudden cliff edge, particularly for young students. Doubting gave us a useful insight into retention, albeit an imperfect one.

Impact of course teams on retention

We carried out a case study approach and worked with ten courses across the three partners. We investigated how these course teams worked with students they saw at risk and supported them.

We identified two broad types of activity. Firstly, activities that course teams carried out to reduce the likelihood of doubting in the first place. Secondly, work with doubters and other students at risk to intervene once doubts or problems occur. If you like both a preventative and remedial approach.

We developed nine specific recommendations and wrote them into a toolkit and recipe cards for course teams to use to develop their own working practices.

• Identify and respond to students at risk (knowing your students)
• Help students to make the transition to being effective learners at university 
• Relationship and communication with staff
• Help students make more-informed decisions about choosing the right course in the first place 
• Improve social integration 
• Improve a sense of belonging to the programme 
• Foster motivation and help students understand how the programme can help them achieve their future goals 
• Encourage students‘ active engagement with the curriculum 
• Ensure that there is good communication about and access to additional student support 

Conclusions

I still believe that there’s a huge amount of richness in both the original “What Works?” report and our HERE Project report and toolkit (3).

• Engagement is core. As I’ve picked out in an earlier post, we designed our learning analytics resource to be based on engagement, not risk of early departure.
• There are early warning signs. Our survey work wasn’t perfect, but student doubters were more at risk of early departure than their peers.
• We still don’t have the perfect intervention for course teams. We know that you need to start with ‘knowing your students’, and we found a range of different activities that all in some way helped, but no magic bullets.
• Too many of the original PAC recommendations still need addressing, but the problems remain fiendishly complex, if anything, more so than in 2008.
• We needed this research to give us the intellectual underpinnings to properly conceptualise our approach to learning analytics. I’d argue that anyone interested in using learning analytics to support student success, needs to really develop their retention and success hinterland.

Postscript

In the summer of 2012, we were running end-of-project workshops for lecturers and support staff. We knew that we had important lessons to share, but like so much in HE, we knew that it wasn’t sinking in. People were interested in our work., they particularly liked the recipe card idea as a way of generating discussion. But it was yet another initiative that needed staff to change the way they worked. At this point, my very brilliant colleague Sarah Lawther went back to a paper she’d uncovered earlier, Mantz Yorke’s “Gold in them there hills…” (4).

And that’s when we really got interested in learning analytics.

 

Footnotes ‘n’ stuff

(1) One of the most awkwardly-titled projects in HE history, primarily because MS Word insists on capitalising immediately after ‘?’ even if you are mid-sentence.

(2) OZGA, J. and SUKHNANDAN, L., 1998. Undergraduate non-completion: developing an explanatory model. Higher Education Quarterly, 52(3), pp. 316-333.

(3) I’m struggling with some aspects of the notion of blogging, this isn’t an ego trip. I promise. This was a real team effort with lots of valuable inputs from colleagues at NTU, Bradford and Bournemouth.

(4) YORKE, M., 2006. Gold in them there hills? Extracting and using data from existing sources. Tertiary and Education Management, 12, pp. 201-213.