Doctoral Thesis on Interoperability and Academic Integrity Related to Digital Assessment

By Guttorm Sindre, April 1, 2022

PhD candidate Aparna Chirumamilla, who defended her thesis in the fall of 2021, researched digital exams, specifically focusing on challenges related to the lack of interoperability between various tools and security against cheating.

Interoperability refers to the extent to which different IT tools can interact, ideally seamlessly, to avoid redundant work related to the combined use of different tools. There are two main goals for interoperability:

• Easy exchange of data
• The ability to seamlessly run one tool from another

A scenario where data exchange between learning tools is useful is when instructors have created exams in Inspera and intend to use the same exams for formative assessments in learning support systems like Canvas or Blackboard the following year. Or they might want to share the exams with instructors at another university using WISEflow for digital exams. In these cases, the ability to export exams from one tool and easily import them into another is preferred, rather than creating the exams again in the new tool. While this can sometimes succeed, many tools may have different implementations of the QTI (Question and Test Interoperability) international standard, resulting in limited success. For example, Inspera has a specific task type called "Place in text," useful for puzzle-type problems (Parsons Problems) in programming. However, if the target tool for importing does not support this type of task, exporting it as QTI becomes futile.

Furthermore, seamless running of one tool from another might involve allowing students to experience the same user interface during exercises and self-tests throughout the semester as they will encounter during the final exam. This seamless integration would entail starting the digital exam system from the learning support system (e.g., initiating Inspera from Blackboard or Canvas) without requiring students to log in again. This would enable students to perform the relevant tasks without re-logging in and have their completion recorded in the learning support system. Similarly, instructors or teaching assistants should be able to move seamlessly between different tools to provide feedback on submitted work. The international standard for achieving this is LTI (Learning Tools Interoperability), but there is still considerable progress needed before such seamless integration becomes a reality.

Chirumamilla's research revealed that the establishment of UNIT, which coordinates national IT tool procurements for the education sector, has increased the potential for a holistic approach to interoperability and the cooperation of many tools. However, there is still a long way to go before tools can fully support the seamless interoperability that is desired. In concrete tool procurements, interoperability often takes a back seat compared to other criteria, such as new functionality and ease of system operation.

Security against cheating is a crucial criterion for digital exams. It might intuitively seem that digital school exams (under supervision) pose a greater risk of cheating than their paper counterparts. The types of cheating possible in paper exams, such as cheat sheets or whispered communication between candidates, are typically feasible in digital exams as well. Moreover, digital exams introduce new threats of cheating via computers, especially in BYOD (Bring Your Own Device) exams. However, Chirumamilla's research suggested that digital exams do not necessarily pose a higher risk of cheating. In addition to new threats, digitalization also enables more anti-cheating measures that would be expensive or challenging to implement in paper exams. These measures include variations in the order and type of tasks, restrictions on timing and navigation between tasks during the exam, and the ability to observe a student's progress keystroke by keystroke.

During the COVID pandemic and home exams, there has been increased focus on preventing cheating. Most of Chirumamilla's research was conducted before lockdowns, so home exams were not a primary theme in the thesis. Nevertheless, the research touched upon this to some extent. Many international institutions use student monitoring during home exams through webcams and microphones, but such systems are not currently employed in Norway. However, it's essential to understand that even such systems aren't a 100% guarantee against cheating, as technically skilled students might find ways to circumvent them. It's crucial not to overly rely on surveillance in the fight against cheating but to focus equally on creating positive attitudes and pedagogical aspects: provide students with better coping mechanisms throughout the semester and reduce anxiety about exams, making it more challenging for students to cheat on assignments.

Further reading for special interest:

Chirumamilla, A., Sindre, G., & Nguyen-Duc, A. (2020). Cheating in e-exams and paper exams: the perceptions of engineering students and teachers in Norway. Assessment & Evaluation in Higher Education, 45(7), 940-957.

Chirumamilla, A., & Sindre, G. (2021). E-exams in Norwegian higher education: Vendors and managers views on requirements in a digital ecosystem perspective. Computers & Education, 172, 104263.

Chirumamilla, A. (2021). Analysis of security threats, requirements, and technologies in e-exam systems. PhD thesis, Department of Computer Science, NTNU, October 2021.
Sindre, G., & Vegendla, A. (2015). E-exams versus paper exams: A comparative analysis of cheating-related security threats and countermeasures. In Norwegian Information Security Conference (NISK) (Vol. 8, No. 1, pp. 34-45).

Sindre, G. (2021). Kan fusk på hjemmeeksamen forhindres? Nordic Journal of STEM Education, 5(1).