Students’ approaches to solving R-FCI tasks observed by eye-tracking method

This study sought to assess the representational format of task options in the representational variant of the force concept Inventory (R-FCI) test, namely its impact on students’ problem-solving approaches. This was done with the help of eye-tracking equipment. 35 high-school students solved four tasks, mainly from the R-FCI test, which sought to assess the student’s understanding of Newton’s 1st and 2nd Law of Motion. As they were trying to solve the problems, their gazes were tracked by TobiiTX300. A comparison between students who provided the correct and incorrect answer was subsequently carried out. The correctly answering students very quickly found the correct solution both in verbal and graph representation. For motion map representation, they usually compared and made decision between two options. The incorrectly answering students did not show any consistent strategy except they paid the least attention to the correct answer. Moreover, two case stud studies of correctly and incorrectly answering students were described.

Students' misconceptions and difficulties when learning science have been in the forefront of scientists' interest for decades.First focus was on mechanics, especially on students' understanding of Newton's Laws, which resulted in creation of The Force Concept Inventory (FCI) (Hestenes et al., 1992).Recently, representational variant of the force concept inventory (R-FCI) was developed by Nieminen et al. (2010).For nine original FCI items two new isomorphic variants were formulated in different representations, e.g.text, graph or diagram.The ability to use multiple representations is an essential scientist's skill (Hestenes, 1996).As Nieminen et al. found in some tasks, students' understanding was statistically significantly different when posed in different representation, so representation needs to be taken into account when one discusses a physics concept.
The main aim of our study was to compare approaches of students who solved tasks focused on understanding of the 1 st and 2 nd Newton's Law concepts from R-FCI test correctly or incorrectly.Furthermore, we were interested in the way how the representation of the task affects student's solution.Observation was provided by the eye-tracking method, which can give deeper insight into students' thinking processes via tracking of participants eyes.
The method is based on the eye-mind assumption (Just & Carpenter, 1980) and is possible to be used for both qualitative and quantitative type of research design (Bojko, 2013).In qualitative approach, one can observe the thinking process of a participant via gaze plot or provide qualitative comparison of different students group using gained heat maps (Kekule, 2015).Quantitative approach is focused on comparison of typical eye-tracking metrics, such as total fixation duration, average fixation duration and number of fixations (Duchowski, 2006) on defined areas of interest of presented materials.
Comparison of correctly and incorrectly answering students came from expertnovice paradigm and is common for research in the education field.There are several typical differences observed between experts and novices.For example, Gegenfurtner et al. (2011) observed that experts are able to collect information from the parafoveal area.In addition to that, they are better able to distinguish between important and irrelevant information from each other, so they typically show shorter average fixation duration and bigger number of fixations on areas relevant for the right solving of a task and vice versa.Moreover, Chi et al. (1981) found that experts can create an internal representation of the task faster than novices.Experts generated more quickly an idea of how the task should be solved and what the solution would be.
In physics education, probably the most studies were interested in mechanics, focused on students strategies when they solve tasks of FCI test, for example Madsen et al. (2012), Kozhevnikov (2007), Ohno (2016).Smith et al. (2010) were interested in a way how students work with textual (conceptual) and mathematical clues when they deal with problems from mechanics.The method has been used for computer testing as well.For example Chen et al. (2014) carried out a study focused on prediction of likelihood of responding to the correct physics concepts from various physics fields successfully.Detailed overview of eye-tracking studies in physics education till 2010 is provided in Kekule (2014).
1 Research Method

Test tasks
Using the eye-tracking method we observed students when solving four tasks from mechanics, particularly three from the R-FCI test (Nieminen et al., 2010) and one was originally created for purpose of the test in the same manner as the three previous ones.All tasks were focused on understanding of Newton's laws concepts; two on the 1 st Newton's law and two on the 2 nd Newton's law.The test basically consists of four different questions, each of them with options in three different representations: verbal, graph and motion map.All task stems appeared in written form.Because of preservation of the confidentiality of the original FCI items we do not publish all tasks which were included in the testing.For an example item see fig. 1.
A spaceship drifts in outer space.The spaceship is subject to no outside forces.At the instant of time t 0 the spaceship's engine is turned on and produces a constant force on the spaceship.The force is in the direction of the motion.& Slavik, 1989).Students were asked to state their preferred way of learning (visual, audio, tactile or kinesthetic) and moreover to state their attitude to conformity in sense of willingness to fulfill tasks given by their teachers.Altogether, students were asked to answer through items from five dimensions of LSI.The questionnaire consists of statements and students express on the 5-point Likert scale the extent of their agreement.

Participants
High school students and students just enrolled in their first university year study took part in the study.Altogether, data from 46 participants were gained.As a valid data only records with more than 70% caught eyes positions were considered and further processed.Records from altogether 35 students have met the threshold.44% male and 56% female; 2 physics teachers, 6 future teacher students in their first year of college study and high school students took part in the research.Data collection was carried out in November and December 2016.

Eye-tracking equipment and testing procedure
The eye-tracker by Tobii was used, particularly TX300 with frequency 300 Hz, which has an accuracy less than 0.5 • of visual angle.The infrared camera was placed under the 23-inch screen of the stimulus PC. Stimuli were presented as a pdf-document.Participant's eyes were positioned at a distance of approximately 70 cm from the center of the screen.A five point calibration and validation procedure was used before the start of the experiment.Eye movements were recorded by Tobii Studio 3.2 and for identification of fixations inbuilt IVT filter were used.Eye movement was classified as a saccade when eye's velocity exceeded 30 • /s.Minimum fixation duration was set to 60 ms.Extra mouse and keyboard were connected to the stimulus PC by which participants handled the PC.The eye tracking session lasted about 8 minutes.During the whole session students were asked to think aloud when it was needed and the sessions were recorded by video camera.After the sessions students were asked to provide verbal retrospective report.First participant solved test version 1, second participant test version 2, etc. so that participants from different groups took part in all test versions.

Data analysis
Data analysis was provided by Tobii Pro3.2 software by Tobii company (tobii.com).

Eye-tracking metrics on defined areas of interest
A comparison of typical eye-tracking metrics on defined area of interest (AOI) is a common analysis in eye-tracking method.As AOIs particular options and stem of a task were defined and as an appropriate eye-tracking metrics total fixation duration mean on an AOI was used.As we provide comparison between two student groups, we do not need to take into account different AOIs' size.An example of defined AOIs is shown in fig. 2. Allocation attention on AOIs is shown for each task and two student groupsthose, who answered correctly (C group) and those who answered incorrectly (INC group).

Attention map comparison
Attention maps can provide detailed insight into students' allocation of attention within defined AOIs, especially larger AOIs such as stems or options in motion map representations.Attention maps presented in the paper were created by Tobii Pro 3.2 software and were based on number of fixations.

Gaze plots
Gaze plots show all fixation of a participant on the screen during the testing.Gaze plots presented in the paper were created by Tobii Pro 3.2 software.Diameter of the fixation circle represents single fixation duration.Lines connecting fixations represent saccades.

Test results
Students needed mostly 3 up to 4 min to complete the test and the whole time ranged from 1:39 to 6:02 min.
Frequency and percentage of correct answers to the test tasks is presented in tab. 2. As we can see, the task "woman pushing a box" was generally the most difficult for students.Correct answers for different representations vary a lot for all tasks.We would point out that verbal representation has not always been the most correctly answered representation.Moreover, students who solved test version 1 provided mostly the least number of correct answers.As it is mentioned above, test versions were assigned to students one by one, so we would expect low effect of group dissimilarity on the total results.In the original test (Nieminen et al., 2010) similarly the task "woman pushing a box" had very low difficulty index, under 0.3.As we can see, tested students were most successful in graph representation of options for this task.The two tasks about the rocket had in the original test difficulty indexes about 0.70 ("rocket after") and 0.40 ("rocket before").Our results show very high variation of the percentages of correct answers especially for the task "rocket before" in different representations.

Choice of an option -attention allocation on defined areas of interest
As eye-tracking method can allow us to follow students' eyes and so roughly their attention, it gives us much more information than only tests results.We can follow their decision making process as well.In order to carry out this, we marked multiple choice alternatives and the task stem as our AOIs (See fig.2) and we were interested in the total fixation duration on the AOIs.Particularly, in the mean of the total fixation duration for two student groups: those, who answered correctly (C group) and those who answered incorrectly (INC group).Results are presented in tab. 3.Both student groups spend the most time on the stem AOIs.For four tasks ("rocket before" in graph and motion map representation and task "astronaut" in motion map representation and "woman pushing box" in graph representation) students in C group spend more time than student in INC group reading the stem.Table 4 shows detailed results, particularly ratio of total fixations duration mean on stem to total fixations duration on all AOIs for both C and INC group.Students in C group on average spend the least amount of time on the stems for motion map representation, whilst for verbal representation they spend about 2/3 of all fixation time on stems.In other words, students in the C group have found solution for verbal representation very quickly, whilst for motion map representation they needed more time.Students in INC group vary in percentage of time spend on stem similarly to students in C group (40-70%), however, we cannot observe any clear pattern in regard to task representation.

Newton's 1 st law concept
When solving two tasks focused on Newton's 1 st law, students in C group paid the most attention just to the correct option -speed is constant -and they spend less time on the other options for verbal and graph representation.For the "rocket after" task they were slightly more interested in the option "speed is steadily increasing" in verbal representation as well.For motion map representation, they were interested in one more option, which suggests that the speed is increasing and later constant (option b).Students who solved correctly a task "astronaut" where interested in other options including increasing speed (option b) as well; however, they paid at least two times less time to the option showing decreasing speed (option b).
INC group of students provided different allocation of attention.For motion map representation, they spent for both tasks much more time on just one option.For "astronaut" task it was "steadily increasing" (option d) and for "rocket after" task "speed is increasing and later is constant" (option b).For graph representation, they were very clearly interested in two options.For "rocket after" task they were interested in all options including decreasing function.For "astronaut" task, they were interested in all options where at the beginning of statement it says "speed is increasing".For tasks in verbal representation, INC group students were interested in one or two options.For both graph and verbal representation, they spend the least time on the correct option (see blue columns in fig.3).
These results suggest that correctly answering students already know the answer after reading the stem and only pick up the correct choice (blue column in fig.3).Or they are able to very quickly recognize which option is absolutely irrelevant as in the case of motion map representation, whilst INC group students' answers can probably be influenced by representation itself.

Newton's 2 nd law concept
For the students in C group we can again observe the same pattern for verbal and graph representations.They are much more interested in the correct option.See for example data for a task "woman pushing a box" in verbal representation (fig.4, left).None of the students answered the task in motion map representation correctly.Students in INC group show different approaches.For tasks "rocket before" they are clearly attracted to option "speed is steadily decreasing" in graph representation, whilst in verbal representation, they paid the most attention to the two other options.The task "woman pushing a box" was for students the most difficult.In motion map and text representation, they paid similar amount of attention to almost all options.The correct answer (option c) suggesting "steadily increasing speed" was the least popular (fig. 4, right).The task in graph representation shows that they were the most interested in just two first options.

Comparison with Finish study
Eye-tracking the R-FCI tasks have been carried out in Finnish study (Jouni et al., 2017) as well.Students show some similar and some different approaches.From Newton's Law concepts point of view, Czech students show similar difficulties like Finnish ones.Firstly, a task "woman pushing a box" was the most difficult for students.Furthermore, for the task "rocket after" focused on 1 st Newton's Law they were interested in all options including word "decreasing", which shows typical misconception.
From representational point of view, correctly and incorrectly answering Finnish students for motion map representation paid the least attention to the option suggesting decelerating motion and preferred to check all the other options.

Reading stem -attention maps comparison
Attention maps can provide detailed insight into students' allocation of attention within defined AOIs, especially larger AOIs such as stems or options in motion map representations.

1 st Newton's law context
When reading the stem of "rocket after" task both student groups paid the most attention to the key information, that engine of the rocket is switched off.Students who correctly answer "astronaut task" were the most interested in information that he or she is not fastened to the spaceship.This pattern is obvious for group of incorrectly answering students only for verbal representation.

2 nd Newton's law context
When reading the stem of a task "rocket before", both student groups paid the most attention to the letters representing key time events.At the task "woman pushing a box", similarly to the task "rocket after", both students group paid the most attention to the statement, that at particular time woman excerpts twice bigger force.

Comparison with Finish study
Finnish students show differences when reading stems based on their stated either correct or incorrect answer.Correctly answering students were the most interested in tasks concerning "rocket" with words "outer space" and "does not affect", while incorrectly answering students similarly to all students in Czech study focused on variable values marked t 0 , t 8 , etc.

Case studies -gaze plots and eye-records
Two participants were included in detailed view on their problems solving.Both solved tasks in test version 3, came from the same school level and their maths and physics grades are at the similar level.They differ in their gender and total gained test score.One participant got the null score from the test, whilst the other got the highest possible score.Therefore, they can represent novice and expert within expert-novice paradigm.However, it is important for the interpretation to take into account that their school performance is assessed to be at the similar level.Analysis of three tasks, each with options in the three different representations will be provided.
Jane, participant no.11, is a 17-year-old female high school student.Her grades from math and physics are at the best level.However, she does not enjoy physics much, but she finds it useful.She stated strong preferences in LSI test for kinaesthetic learning style, which means need of movements during learning or to be engaged in real life experience during learning.Projects, field trips, visiting science learning centres are the appropriate school activities for these types of learners.She answered all tasks incorrectly.
Peter, participant no.23, is a 16-year-old male high school student.His grade from math is at the best level, from physics is one level worse.He likes physics and finds it useful.He stated strong disagreement to the idea that he would like to do something with his hands during learning (e.g.build something, to make a model by hands, etc.) On the other hand, he prefers to learn by his own experiences.He answered all tasks correctly.

Verbal representation
Tab. 5: Gaze plots for the two high school students for a test task with options in the verbal representation -task "rocket after" in verbal representation Jane Peter Both students paid the most attention to the option, which they in the end chose.Jane read first the stem, then each option.After reading d) she returned back to c), a), again she was reading c), d).Then Jane took only one fixation on the correct option e).In the end she took several fixations on the option c) alternating with picture of the rocket in the stem.
Peter read the stem carefully and then option a) about at first constant and then decreasing speed.After that, he returned to the first row of the stem and again paid attention to option a).He read the question in the stem and then he went through all the options and took a fixation on the empty part.After that he took a short look on each option and went back to the stem question.After that, he read the question and option e).Before the end of the record, he glimpsed at a picture of rocket in the stem and confirmed e) as the correct option.Based on the gaze plot, we can notice that Peter was reading option a) for longer time in contrast to the other options.But this happened only at the beginning together with reading the stem.

Graph representation
Tab. 6: Gaze plots for the two high school students for a test task with options in the graph representation -task "woman pushing a box" in graph representation Jane Peter Both students took several fixations on each graph option.Again, Jane showed the least number of fixations on the correct option c.She started reading the stem and then she took a look at the first graph, particularly, at the variables along the axes.After that, she went through the other options; however, she was focused only on vertical axes, variables there and the value -initial speed.Then she returned to the stem, particularly to the part about the initial speed value.She went through options a) and b) and then she roved round the stem for longer time.
She went through all the options in the column down and up and then she switched between d) and e), which both depict constant speed, several times.Finally, she went up shortly through all the options, read the stem and chose the option e).
Peter read the first sentence of the task stem, i.e. the information, that a woman is pushing a box by a constant force.Then he glimpsed at graph a) and three times reread the first sentence.After reading the whole stem and the stem question he looked at graph b), again at the stem question and several times he was reading the stem and looking at graphs a) and b).When he reread the stem question "which of these graphs" he looked at the other graphs, i.e. c), d) and e); at graph c) he was interested in graph variables.Again he returned to the stem and repeatedly he went through graphs c), d) and e).After that he went up through all the option, again down and in the end he looked at and chose the option c).

Motion map representation
In summary, based on the gaze plot in tab.7 we can notice that Peter easily deciphered decelerating motion, and he did made only several fixations on the three motion maps.Then he carefully counted which of the two remaining options represents the correct answer.As we can notice, for both options, he is interested in the Again he returned to the stem, to the key information that 'engine is switched off' and after that his eyes mostly draw a rectangular defined by front and end parts of options b) and d).Then he stated option d) as his answer to the stem question.
Jane seems not to compare any two options with each other.After reading the stem, she was interested in option b) shortly, then in the front part of d), she carefully examined option e) and then again she returned to the front part of d).She was again interested in the stem and took a look at option a).After that, she once fixated the very front part of each option and moved to the end part of option d).Again she went through the whole motion map in the e) option, look at the stem question and chose e) as her answer.
In contrast to Jane, Peter provides more dense fixations on words.Jane returns to her favourite words more and the gaze plots then creates structures like "grapes", which we do not observe with Peter much.

Conclusion
The most difficult task was the task "woman pushing a box" focused on understanding of Newton's 2 nd law.Similar results were showed by the original testing of R-FCI (Nieminen et al., 2010).The other tasks show high variation in different representation.
"Rocket before" task was the only task where students provided the most correct answers for versions with options in verbal representations.Obviously, the representation cannot be an indicator of student's real conceptual understanding.
Students spent the most time on the stem AOI, which is in agreement with considering FCI test as the quick choice test.Students who answered tasks correctly spent the least time from the whole amount of time on reading the stem for motion map representation, whilst for verbal representation, they took for reading stem about 2/3 of the whole time.Students who answered tasks incorrectly do not show any similar pattern.
Allocation of attention to each offered option shows that correctly answering students already know the answer after reading the stem and only pick up the correct choice, especially for verbal and graph representation.Or they are able to very quickly recognize which option is absolutely irrelevant as in case of motion map representation, where they usually were interested in and compared two options.In the correct one, which presented monotone function of speed depending on time and in one, which was either at the beginning or at the end of time interval similar to the correct option.
Students who answered incorrectly provide different allocation of attention.They do not show any other consistent strategy in solving tasks with motion map representation.Sometimes they were equally interested in three or four options, sometimes they paid larger amount of attention to only one option.For tasks in verbal representation, they were interested in one or two options.For both graph and verbal representation, they spend the least time on the correct option!Based on cased studies there were differences in approach of the correctly and incorrectly answering students.Particularly, they went through options in graph and in motion map differently.The correctly answering student was more interested in the whole area of graphs, whilst the incorrectly answering student focused her eyes mainly on vertical axes.For motion map representation, we can easily observe a pattern of comparison of the two the most probably correct options by the correctly answering students, whilst incorrectly answering students seem only to estimate what kind of motion is presented in the options.
Fig. 1: Example of an item from the R-FCI test in verbal representation: Rocket before stopping the engine

Fig. 3 :
Fig. 3: Mean fixation duration spend on defined AOIs of "astronaut" task in verbal representation.On the left data for the C group, on the right for the INC group of students

Fig. 4 :
Fig. 4: Fixations duration mean spend on defined AOIs of "woman pushing a box" task in verbal representation.On the left data for the C group, on the right for the INC group of students

Tab. 7 :
Gaze plots for the two high school students for a test task with options in the motion map representation -task "rocket before" in motion map representation Jane Peter beginning and the end of motion maps, so that he could easily compare differences.Based on the record of his eye movements, we can observe that after reading the stem and looking at option b), he was first interested in option d), particularly in the end and then in the front part, then he moved to the front part of option b) and to its end.After that first examination, he looked alternately at end parts of options b) and d) and then at front parts of options b) and d) again alternately.See transitions in the left picture in tab.8. Tab.8: Gaze plots of Peter for task's options in motion map representation during solving task "rocket before" in the motion map representation after 0:19 s of watching after 0:24 s of watching Then he compared values of the two options b) and d) in the middle part of the motion map.See transitions in the right picture in tab.8. Then he again read the stem and started to be interested in option a) shortly and examine option e).
Tab. 4: Ratio of total fixations duration mean on stem to total fixations duration mean on all AOIs for both C and INC group *ratio of total fixations duration mean on stem to total fixations duration mean on all AOI