Student Learning and Assessments in the Age of AI

Jul 23

Jul 23 Student Learning and Assessments in the Age of AI

By Carolina Kuepper-Tetzel

In May I attended an all-day ‘AI in Assessment’ workshop with a focus on Psychology degrees in Higher Education in London. Researchers from the UK and Ireland shared insights on students’ attitudes on AI, rethinking assessment approaches, and general considerations on navigating technological progress. I left the workshop thinking: I need to write this up because these insights apply to the wider educational sector! In today’s post, I feature research from Dr Mark Carrigan (University of Manchester), Dr Patricia Gasalla Canto (Cardiff University), Dr Laura Contu (University of Bristol), Dr Michael Smyth (University of Bristol), Dr Kirsty Dunn (Lancaster University), Dr Lara Warmelink (Lancaster University), and Prof Oliver McGarr (University of Limerick).

Keynote speaker Mark Carrigan set the tone by highlighting that the crisis of AI in education is a crisis of trust. When you think about it, the introduction of AI led to an increase in mistrust among all parties involved: Students don’t trust each other, staff don’t trust each other, staff don’t trust students and vice versa. This mistrust has also been revealed on a greater institutional level in work by Patricia Gasalla Canto and colleagues with students revealing institutional trust issues.

The crisis of mistrust led to two things:

We immediately entered ‘cheater detection mode’.
We generated preventative solutions to AI.

However, both approaches fail to realistically address the problem and hinder meaningful learning about and with new technologies. Carrigan also urged educators to approach this crisis in a more diagnostic way and explore why students may be using AI in non-optimal ways by taking external factors into consideration (e.g., time/resources poverty, caring responsibilities). Oliver McGarr suggested instead that we need to move away from the cheater rhetoric and instead explore innovative ways to adapt to changes in education that are brought on by technological innovations. He outlined the four phases of technology governance that help explain our reactions to novel technologies in general, but that can specifically be applied to AI in education, too:

Phase 1: Restriction & bans

What it looks like: Prohibit the use of AI

Examples: Invigilated assessments; restrict access to websites

Phase 2: Detection & policing

What it looks like: Monitor student submissions and address AI issues

Examples: Apply criteria to detect inappropriate AI use (e.g., referencing, unusual content/analyses)

Phase 3: Retrofitting assessments

What it looks like: Modify traditional assessments to make them less prone to AI

Examples: Design assessments that require more creativity and critical thinking; increase difficulty of questions

Phase 4: Re-imagining assessments

What it looks like: Redesign assessment approaches that meaningfully integrate AI

Examples: Shift to real-world project-based, experiential learning and use AI as collaborator; assess the process not the outcome

McGarr used The Locomotive Act 1865 (aka Red Flag Act) as an analogy to demonstrate humans’ resistance to change in face of new technologies. The law was introduced after lobbyists from locomotive industries influenced the government to strictly regulate automobiles. It was enforced under the pretext of road safety but was essentially an attempt to stifle technological progress that provoked fear in losing business. The law restricted the speed of cars to 2-4 mph and required at least three people present with two of them traveling in the car and one of them walking in front of the car holding a red flag. I found this analogy eye-opening because such approaches to technological progress may appear sensible and acceptable at the time, but they are ineffective in stopping technological development. Looking back at such a law with the knowledge we have today, makes us chuckle at its ridiculousness. It is quite likely that future-us will experience similar feelings when looking back at our attempts to navigate GenAI today.

The four stages of technology governance do not necessarily happen in chronological order, but I would say that the experience with AI in education started with the first two stages and looking at the newest AI guidance update at my institution (University of Glasgow) we are moving more toward stages three and four. For assessments grounded in learning outcomes that emphasize knowledge retention and unaided demonstration of knowledge, stage one will remain appropriate. Thus, the goal is to create a balanced combination that allows students to develop crucial foundational knowledge on which skills can be built and to integrate new technologies in a supportive way that does not outsource essential human cognitions.

The call to move away from a ‘cheater mode’ has also motivated Laura Contu’s and Michael Smyth’s research. Contu explained that old framings of “How do we make assessments GenAI-proof?” or “How do we catch students who use it?” are unsuitable and instead suggested a new framing: “How do we teach students to think with GenAI?” Contu and Smyth designed an 11-week GenAI workshop series with 30-min content and activities each week. They conducted focus groups before and after the intervention to evaluate changes in students’ views towards AI. The students were master conversion students. After the second focus group, all students completed an GenAI critique assessment. For the assessment, students were asked to generate one section of their report using GenAI and then write a critique about that section. Here I’d like to focus on the students’ perceptions towards AI before and after the intervention.

In the focus group prior to the intervention, she found that students had a certain level of mistrust in AI in relation to its accuracy and that there was uncertainty around institutional rules (e.g., when is it OK to use AI and when not). There were also concerns surrounding how relying on AI could negatively affect their skills. A point that was echoed in research by Canto and colleagues who showed that students were aware of potential issues with critical thinking and becoming too dependent on AI.

The focus group after the intervention revealed interesting insights:

An in-depth exposure to GenAI led students to realize that to obtain high-quality output you need to invest considerable time and effort. Some even concluded that it may be easier to do the work themselves. In addition, students noticed shortcomings with using GenAI which increased their mistrust. This aligns with research by the group around Canto who found that familiarity with GenAI does not necessarily translate into more acceptance of it. Interacting with new tools can reveal issues and that can help students make better informed decisions.
Students were eager to learn more about GenAI and wanted to know how to use it in more sophisticated ways. My own interpretation of this finding is that the curiosity of learning more about this technology requires knowledge application and a better understanding of the technology. This goes beyond ‘cheating’ and aligns more with phase four of the technology governance framework discussed above.
Finally, there was a call for clearer institutional GenAI guidance. Students were sometimes confused about what is allowed and what isn’t. Certain GenAI use being allowed for some assessments, but not for other, albeit similar, assessments seems unhelpful. The presentation by Kirsty Dunn and Lara Warmelink (Lancaster University) on the day highlighted the same issue. They described a more radical solution to this problem: They moved from an ‘AI banned’ to an ‘AI allowed’ model. To support this shift, they updated all learning outcomes, extended their skills training, and reduced courses without non-invigilated assessments. Their main argument for this shift was to increase employability in students by fostering AI literacy. At my institution (University of Glasgow), we went through a similar process of re-defining approaches to GenAI and ended up with a 2-scenario approach (supervised (GenAI use not permitted) versus unsupervised assessments (GenAI use permitted and acknowledged)). Again, the technology governance framework quite accurately reflects the different phases higher education has gone through in the past couple of years.

An additional take-away from Contu’s talk was that GenAI literacy should not be a one-off session or a couple of slides as part of a lecture. Rather, students need to gain some guided, hands-on experience with the tool to obtain a better understanding of its benefits and pitfalls.

The workshop talks were followed by an audience discussion led by Dr Debra Malpass (British Psychological Society (BPS), Director of Research, Education, and Practice). The one statement that stuck with me from that discussion is this:

Make the thinking process rewarding and the main component we are looking for in assessments!

That point very much resonated with me. When re-imagining assessments for students, the joys of thinking should be at the forefront as a criterion. How can we engage students’ minds meaningfully so that they learn from the process? The idea that assessment should focus on the process rather than the product was also raised by Canto in her presentation. Her team has applied the Problem-Based Learning (PBL) approach in their Year 1 Psychology teaching to tackle this. The idea of PBL is that students work in groups to find solutions to defined problems that are linked to the course content. The instructor assumes the role of a facilitator and guides students by monitoring progress and providing thinking prompts. Importantly, the tutor does not offer solutions, but they do provide feedback.

PBL is used prominently in STEM disciplines but has also found its way into Law teaching. As part of our Science Communication course in Psychology at the University of Glasgow, my colleague Dr Chiara Horlin developed a series of PBL scenarios. Below are two PBL examples from that course:

A) The ‘University Mythbusters’

Problem: “Student services have noticed myths like 'you either have motivation or you don’t' are common. They want to challenge these using psychological research.”

Task: 1) Select a common 'myth' and a relevant psychological theory or finding, 2) Decide which medium would suit a short campaign for this demographic (poster, short video, Instagram carousel) and 3) Create a tagline or call-to-action suitable for peer networks.

B) The Psychology Behind Scare Pranks

Study: Investigations into why people laugh after being scared reveal a close link between fear and humor in the brain.

Task: Design a science communication campaign. Consider who the audience is, what you want to tell them, how to communicate the information effectively, and why they should care about it.

From the talks and discussions, the future in education conceptualizes assessments as professional formation (McGarr) that ignites the joys of thinking and enables learning through the process of engaging with the assessment. To achieve this alongside GenAI will require some re-imagination of assessments. You may be wondering: Well, should that not have been the way all along? The answer to this is yes, but it is sometimes easier to stay within our comfort zone particularly if time is scarce anyway. However, revolutionary developments (i.e., Gen AI) can bear opportunities for innovative progress in learning and teaching because it pushes us to re-think existing approaches and design novel ones.

(cover image by Keira Burton via Pexels)

Jul 23 Student Learning and Assessments in the Age of AI

Jul 10 Time Management: What is it, who has it, and can you improve it? (Repost)