BASE Mental Effort Monitoring Dataset

Description

This study was designed to investigate how mental effort in reading and understanding programs of different complexity can be measured by a set of sensors placed in the programmers (subjects) that participated in the experiment. We used the experimental setup of one of the experiment campaigns of the BASE initiative, which covers a comprehensive set of sensors including EEG with a 64-channel cap, ECG, EDA, and eye tracking with pupillography. The signals from all these sources are synchronized in a common time base to allow consistent cross analysis.

The study included 30 participants and all the subjects had experience in the Java programming language, selected for this study. The screening process included an interview to assign a programming level skill in Java to each participant as Intermediate, Advanced or Expert.

The study was approved by the Ethics Committee of our organization, in accordance with the Declaration of Helsinki, and all experiments were performed in accordance with relevant guidelines and regulations. Informed consent was signed by all participants. The anonymized data from this research will be available upon request to the authors.

3 small Java programs having different complexity were carefully designed to keep consistency in the programming style and to avoid that math or algorithm pose extra difficulties to the participants, not directly related to the code complexity:

– Program C1 counts the number of values existing in a given array that fall within a given interval using a straightforward loop.

– Program C2 multiplies two numbers using the basic algorithm where every digit from one number is multiplied by every digit from the other number, from right to left. The numbers are given as strings and the algorithm includes converting the strings to byte arrays.

– Program C3 seeks occurrences of an integer cubic array inside a larger cubic array, trying to find the larger occurrence of the smaller one inside the larger one. The algorithm for this program has a high cyclomatic complexity having many nested loops.

The table below presents a summary of the complexity metrics of the three programs. In C1 and C3 the algorithm is coded in one function, while in C2 the algorithm is spread across two functions, possibly making C2 easier to read than C3, in spite of both having a comparable number of lines of code.

All the participants performed the experiments in the same room, without distractions, noise or presence of people unrelated to the experiments. The protocol includes the following steps, performed on the screen of a laptop:

1. Empty grey screen with a black cross in its center during 30 seconds. It serves as baseline phase in which the participant does not perform any activity.
2. Screen with a text in natural language to be read by the participant (60 seconds max.). This step serves as reference activity (different from code understanding) for the purpose of data analysis.
3. Empty grey screen with a black cross for 30 seconds.
4. Screen displays the code in Java of the program to be analyzed for code comprehension (C1, C2, C3). This step last 10 minutes maximum.

The protocol scheme is presented below.

Parameters

Time series

• EDA – Eletrodermal Activity
• ECG – Electrocardiogram
• PPG – Photoplethysmogram
• ICG – Impedance Cardiogram
• EEG – Electroencephalogram (64 channels)

Eye tracking data

• Raw data position
• Pupil diameter (left and right eye)
• Cornea reflex position (left and right eye)
• Point of regard (gaze data / left and right eye)
• Head position / Rotation

Meta data

• Screening evaluation data
• NASA-TLX evaluation data

Annotations

• Experts code annotation

Aux data

• Images presented to the volunteers