Observing Students’ Problem Solving Strategies in Mechanics by the Eye-tracking Method

The paper is focused on observation of students’ problem solving strategies when solving concept cartoons tasks testing their understanding of the 1st and 2nd Newton’s law. Students’ solving process was recorded by an eye-tracker and, based on their tracked eyemovements, analysis of their approaches was provided. Students solved tasks from the R-FCI test as well. Detailed analysis of the solutions of four high-school students, one college student and one teacher was provided.

For decades, difficulties and misconceptions which students have when they deal with Newton's laws have been identified, especially via The Force Concept Inventory (Hestenes et al., 1992). Moreover, representational variant of the inventory was developed by Nieminen et al. (2010). Nowadays, concept cartoons (Naylor & Keogh, 2010) as a task form are widely used in the area of education not only for testing, but particularly in the teaching praxis as an appropriate stimulus for the discussion among students. In addition, these tasks are attractive for students because of visual representation of scientific ideas with many visual elements, minimal text and a dialogue form that we can't find in the classical form of test tasks; therefore, concept cartoons are rather similar to learning materials.
Deeper understanding about students' thinking processes can be provided with the eye-tracking research method. The method is possible to be used for both qualitative and quantitative type of research design (Bojko, 2013). In the qualitative approach, one can observe the thinking process of a participant via gaze plot or provide qualitative comparison of different students' groups and gained heat maps (Kekule, 2015). Quantitave approach is focused on the comparison of typical eyetracking metrics on defined areas of interest of presented materials, such as total fixations duration, average fixations duration and number of fixations (Duchowski, 2006). Average fixations duration informs about information extraction difficulty or about an increased interest in an object. The number of fixations can for example indicate the level of expertise (the lower the number the more expertise) or it can inform about search difficulty. Typically, experts show shorter average fixation duration and bigger number of fixations on areas relevant for the correct solution and vice versa (Gegenfurtner et al., 2011).
Detailed overview of eye-tracking studies in physics education till 2010 is provided in (Kekule, 2014a). In physics education, probably the most studies were interested in mechanics, focused on students strategies when they solve tasks of the FCI test (for example Madsen et al. (2012), Kozhevnikov (2007), Ohno at al. (2016)). Smith et al. (2010) were interested in the way how students work with textual (conceptual) and mathematical clues when they deal with problems from mechanics. The method has been used for computer testing as well. For example, Chen et al. (2014) carried out a study focused on the prediction of the likelihood of responding the correct physics concepts from various physics fields successfully.
In this article, we present a case study where we, with the help of the eye-tracking research method, qualitatively assess the strategy of solving a few problems from mechanics focused on Newton's laws of motion.
The aim of our research is a detailed analysis and a comparison of solution strategies of several qualitative tasks aimed at some misconceptions in dynamics. On the basis of the conducted analysis, we also want to explore the possibilities of the use of an eye-tracker for the examination of visually rich tasks. For the purpose of this case study we chose six students -three experts and three novices.
1 Research Method

Test tasks
In this study, we analysed students' problem solving abilities in five tasks focused on understanding of the concepts concerning Newton's laws. Three tasks were adopted from the R-FCI test by Nieminen et al. (2010), one task was similar to the three previous ones and one task was in the form of concept cartoon.
We chose the tasks concerning both horizontal and vercital motion because the development of ideas about forces and motion take place mainly in the context of learning about these types of motion. In tasks there were included some common misconceptions about the relationship between forces and motion which have been identified by many researchers (Mandíková & Trna, 2011;Driver et al., 2003). The most common misconceptions in this domain are (Driver et al. 2003: p. 149): 1. If there is motion, there is a force acting. 2. If there is no motion, then there is no force acting. 3. There cannot be a force without motion. 4. When an object is moving, there is a force in the direction of its motion. 5. A moving object stops when the force acting on it vanishes. 6. A moving object has a force within it which keeps it going. 7. Motion is proportional to the force acting. 8. A constant speed results from a constant force.
Children develop their ideas about the natural world around them from early childhood on the bases of their sensory experiences. Research studies indicate that children have ways of constructing phenomena which are coherent and fit with their domains of experience. However, their ideas may differ substiantially from the scientific view. Studies also indicate that these notions may persist into adulthood despite formal teaching. By means of the eye-tracker, it is possible to follow which misconceptions the students still fix more often in older age and, conversely, to which they pay almost no attention.

R-FCI and additional task
Three tasks were adopted from the R-FCI test (Nieminen et al., 2010) and another one was originally created for the purposes of the test similar to the three previous ones. All tasks were focused on understanding of the 1 st and 2 nd Newton's law. Each item offered alternatives in one of the three following representations: verbal, graphic or motion map. Only two items of the R-FCI are presented here ( fig. 1) in order to preserve the confidentiality of the original FCI items. The task order was chosen, so that the same task in different representations was not expressed twice successively. All task stems appeared in written (verbal) form.
Rocket before stopping the engine Rocket after stopping the engine As mentioned above, at the end students solved one task in the form of a dialogue (the tasks of Ball or Skateboard in fig. 2). This task was presented to students in the form of concept cartoons, which are cartoon-style drawings including a range of viewpoints relating to science situations. They have the form of a multiple-choice question, where cartoon characters (usually three or four) put forward alternative viewpoints about science concepts.

Fig. 2: Concept cartoons Ball and Skateboard
Concept cartoons as an innovative and learning strategy were developed by Stuart Naylor and Brenda Keogh (2010) in 1992. The statements in the bubbles include widely held misconceptions and alternative conceptions. Only one scientifically acceptable explanation usually exists among the statements raised by cartoon characters. However, other statements are not implausible, they are often based on students'experiences (most frequent misconceptions), so that students are likely to see many of the alternatives as credible. If learners are not confident in expressing cartoons characters'ideas, they might express their own thoughts. That's why a blank speech bubble is also included with the text "You are wrong. I think that. . . ".
The concept cartoons can be used in various ways and in a wide range of settings (Naylor & Keogh, 2010). They are often used as a stimulus for discussion, for identifying areas of misconceptions. They are valuable to the learners (and to their teachers too) for clarifying what ideas students hold. This learning strategy is very effective in eliminating the misconceptions. Concept cartoons are also defended by numerous researchers for challenging critical thinking and developing conceptual thinking.
We used the concept cartoons to find out about the ideas that the participants of our research hold, to see if there are any similarities in their strategy of solving the problems and to identify possible differences from the classical test tasks (see the Rocket task). Both concept cartoons we used were also focused on understanding of the 1 st and 2 nd Newton's law. Alternative statements were based on a research identifying common areas of student's misunderstanding (Driver et al., 2003;Mandíková & Trna, 2011). Students always chose only one statement that was scientifically acceptable for them. Each of the concept cartoons included a photo illustrating the situation and there were also photos of pupils expressing their statements. Moreover, each task included other visual elements such as headings, a link to the source of the image, a note, tiny legends of photos, illustrative photo of Albert Einstein etc.
In the task Ball, statement A is the most acceptable. The statements B and C come from the misconceptions no. 2 and 7 (see chapt. 1.1); students often think that the force at the beginning of the motion is strong (as the ball left hand) and diminishes as the ball rises. As the motion stops, there is no force acting on the ball.
In the task Skateboard, the statement C is the most acceptable. The statements A and B come from the misconceptions no. 1 and 6 (see chapt. 1.1); students often find it difficult to understand that an object will keep moving with a steady speed if there are balanced forces acting on it and that when a pushing force stops acting, the object will begin slowing down promptly. Learners often think that an object "is losing" some force gradually if we stop exerting the force (similarly to the task Ball). However, if the stopping of the skateboard were caused by canceling of the acting force, the skateboard would stop immediatelly. The stopping of the skateboard is caused only by friction.
In the task Rocket, students should state what rocket velocity will be when the rocket engine is switched off. The statement e) is the most acceptable. The statements a)-c) come from the misconceptions no. 5 and the statement d) comes from the misconception no. 6 (see chapt. 1.1).

Participants
During autumn 2016, 34 participants were tested and tracked by eye-tracker. 11 students' gaze-tracking data was so incomplete that it could not be used in the study and valid data from 23 participants were used for providing research results. 21 participants were high school students or students in the first year of their college study. 2 participants were physics teachers working at high schools in Prague.
1.3 Eye-tracking equipment and testing procedure

Equipment
Eye-tracker by Tobii was used, particularly TX300 with the frequency of 300 Hz, which has an accuracy of less than 0.5 • of visual angle. The infrared camera was placed under the 23-inch screen of the stimulus PC. Stimuli were presented as a PDF document. Participant's eyes were positioned at a distance of approximately 70 cm from the centre of the screen. A five point calibration and validation procedure was used before the start of the experiment. Eye movements were recorded by Tobii Studio 3.2 and for identification of the fixations an inbuilt IVT filter was used. Eye movement was classified as a saccade when eye's velocity exceeded 30 • /s. Minimum fixation duration was set to 60 ms. An extra mouse and keyboard were connected to the stimulus PC by which participants handled the PC.

Procedure
The eye tracking session lasted about 8 minutes including the calibration process. Before each task a short instruction about the following task was presented, so that students had enough time to relax before they started to solve it. Concept cartoon tasks were presented always at the end of the test. During the whole session students were asked to think aloud when it is needed and the sessions were recorded by video camera. After the sessions students were asked to provide verbal retrospective report.

Results
The characteristics of the individual participants, their test scores and the correctness of task (Skateboard or Ball) solutions are listed in tab. 1. Durations of the solutions and the analysis of the solutions based on the heat maps were presented (Hejnová, Kekule, in print) at the Contemporary Trends in Physics Teacher Training 8 conference in April 2017. Here we want to focus on a more detailed description of the strategies of the best and the worst performing students and of the other participants of our research (two teachers).
Tab. 1: The characteristics of the individual participants, their test scores and the correctness of concept cartoon tak solutions woman college student * 4 -A * Future physics teachers in the first year of their college study. * * Test score of 4 tasks of the R-FCI test. * * * Correct answers are marked in bold.
In our case study we will work in the expert-novice paradigm (Kekule, 2014a) assuming the expert and the novice have different approaches to problem solving. Whether a student was considered an expert or a novice, was decided on the basis of detected characteristics of the research participants.
As can be seen from tab. 1, all five assigned tasks were solved correctly by participants P31, P54 a P47. They were one teacher and two college studentsfuture mathematics and physics teachers (each of them solved a different concept cartoon). We consider these participants as experts and the detailed description of their strategies of task solving will be descibed describe in chapt. 2.1.
The students with worse test performance are considered as novices. We will focus on the analysis of their task-solving strategies in in chapt. 2.2 in detail. College student P39 gave the weakest performance; he solved none of the assigned tasks correctly. A detailed analysis of the solution strategy will be carried out also for college student P27, who solved only one task in the test correctly and also failed to correctly solve the concept cartoon "Ball". Moreover, it took him approximately three times longer to arrive to his solution than it did the best performing student (P47). For further detailed analysis we chose also high-school student P46 who received only one point in the test and solved the concept cartoon "Skateboard" incorrectly. Two other students (P35, P44) who gave similar weak performances in the test were not chosen for the detailed analysis, because we wanted to qualify as novices mainly students who gave bad performance in both the test and the concept cartoon. For example, college student P44 chose in the concept cartoon Skateboard answer D "You're wrong. I think that. . . "; it is therefore not possible to express in detail the particular answer she had in mind.
In the following text, we will discuss in greater detail expert-novice differences in the solution strategies of both concept cartoons. Firstly, attention will be given to the general solution strategies and, secondly, we will comment on the strategies with regard to physics aproach. Regarding the general solution strategy, we will specifically monitor: 1. the total time of solving the task; 2. the number of readings of the assignment before choosing a solution; 3. the number of fixations of pictures and other perceptually significant elements.
From the point of view of the physics approach, we will notice in particular: 4. the number of fixations of key words or phrases relevant for the right solution of a task; 5. the number of fixations of misconceptions.
In case of the students who solved the concept cartoon "Skateboard" we will present for comparison the procedure of solution of one task (the Rocket) from the test, which focused on a similar topic.

The best performing participants 2.1.1 College student P54
From a relatively small number of fixations (see gaze plot of the task "Skateboard" in fig. 3) it is clear that the student came to her decision fairly quickly (solving of the task took her approximately 39 s.). From the video record of her procedure of task solving we can observe her strategy well. After reading the introductory text in the task, she first fixated the illustrative picture of a boy on a skateboard and then she read the statements of the individual cartoon characters while fixating the cartoon characters'photos after reading each statement. After the first reading of all the text she returned to the statement A that she had been fixing for longer periods of time (in particular the phrase "he does not deflect from the ground"). Then she returned immediately to the correct statement C and she was fixing the key concept "friction force" for shorter periods of time.
For this student, we see typically shorter fixation durations, quick elimination of incorrect statement (B) and also a fairly quick decision between the incorrect statement (A) and the most acceptable statement(C). 1 The student also paid quite close attention to the pictures, especially photos of children.
As for her solution procedure of the task of the Rocket (fig. 3), a similar solution strategy as for the task of the Skateboard can be noticed from video record. After carefully reading the assignment of the task and the first four options, she returned to the key information in the question "after turning off the engine". Then the student eliminated the first two possibilities and quickly chose the correct answer (E) from the remaining three answers. The picture of the rocket was fixated only marginally.

Teacher P31
This participant very carefully read all the text (we can observe a great number of fixations of all statements, see gaze plot of the task "Skateboard" in fig. 4) Solving of the task took him approximately 57 s (which is the second longest time in the monitored group). From the video recording it can be seen that after careful reading of all the texts he decided immediately between the statement C, which can be considered the most acceptable, and the last option D (i.e. "You're wrong. I think that. . . "). From gaze plot of the solution of the task there is apparent more frequent fixation of the key concept "friction force". 1 In order to avoid possible complicating of the situation by various considerations, we don't label an answer as correct, but as (from a scientific point of view) "the most acceptable". In the text, however, we will for simplicity use the designation "correct answer".
For this teacher-expert, a very fast elimination of the two incorrect solutions (A and B) is typical. It is worth noting the teacher's relatively significant attention to graphic elements, in particular illustrative picture of a boy on a skateboard.
As for solving of the task "Rocket", like the student-expert P54, the teacher first carefully read the assignment, while fixating the key phrases "engine off" and "after turning off the engine" very soon. When browsing the various options he returned to those key words again several times. Then he was choosing between the last two options, i.e. "the magnitude of the velocity is constantly increasing" (D) and "velocity is constant" (E), of which he chose the right one. The student also fixated the picture of the rocket more often than the student-expert P54. If we compare the gaze plots of the Rocket task solutions in fig. 3 and fig. 4, we can notice the almost identical solving strategies and fixing of the same places in the text to a similar extent.

College student P47
From the gaze plot of the task "Ball" (fig. 5) it is clear that the student fixated the correct statement most often and also for the longest period of time, she paid significantly less attention to the remaining two options. Solving of the task took her approximately 57 s (which was the second shortest time in the monitored group). The student-expert first read all the texts quite quickly; it was probably a quick familiarization with the task. During the second reading she was more careful and during the third reading she already fixated only the correct statement and did not pay any attention to the others. She did not fixate the incorrect idea in the statement B ("force of the hand") at all (she perceived it more likely only marginally (in the so-called area of blurred vision)), she quickly ruled it out, and then did not return to it again.
Again, for this student-expert is typical rapid elimination of incorrect solutions and also minimal fixations of the pictures. On the basis of the gaze plot of task solving ( fig. 6), it can be stated that the number of fixations is in the case of this student-novice roughly the same as for the student-expert. Also, the student-novice reached her solution relatively quickly (the solution took her approximately 31 s), with no longer hesitation over the choice of her answer recorded. After the first reading of all the statements, during the 9(2), 2018, p. 102-116 Fig. 6: Gaze plot of the "Skateboard" concept cartoon and the "Rocket" task (student P39) second one, the student-novice returned immediately to the statement B which she identified as correct, while she did not hesitate at all between the other variants. The student fixated the key concept "friction force" during the first reading for a longer period of time, but in further reading she did not returned to it again, probably because she did not consider this answer (C) as the correct one.
For this student-novice, a smaller number of fixations and shorter durations of fixations, together with a quick selection of an incorrect statement (B) is typical. The student did not pay almost any attention to the pictures.
As for the student's strategy of task 2 solving, from the video recording it is possible to observe a longer hesitation between two incorrect options, while the student fixated for a very long time the word "constant" (i.e. the rocket will move at a constant speed if the rocket motor exerts constant force). When she was making her decision she often returned to the key phrase "motor, which acts on the rocket".
Incorrect solutions of the two tasks show that the student obviously has a typical misconception that for uniform (linear) motion it is necessary for some force to act on the body. The fact that the student did not hesitate too long in solving of the task "Skateboard" with the answer, probably indicates that she was fairly sure of her answer, which, however, appear to be based on a misconception.

College student P46
From the gaze plot of task solving ( fig. 7) it is clear that the student has devoted minimal attention to the most acceptable answer (C) and he also fixated the key concept "friction force" minimally. He paid the greatest attention to the task stem and to the second option (B), which he ultimately chose as his answer (when he had returned to it for the fourth time).
This student-novice was solving the task for a longer period (49 s), he hesitated in choosing his answer and he more often fixated some photos of individual characters.
As for the student's solving strategy of the task "Rocket" 3 , from video recording it is possible to see his repeated reading of the task stem and frequent fixing of the words "at time t 0 the rocket motor is switched on, which operates" and "in time t 8 the rocket motor is switched off". The student paid much less attention to the issue 9(2), 2018, p. 102-116 Fig. 7: Gaze plot of the "Skateboard" concept cartoon and the "Rocket" task (student P46) of what is going on with the speed of the rocket in the interval t 0 -t 8 . Therefore, he probably chose answer B (the magnitude of the velocity initially increases and then is constant). The incorrect solution of this task suggests that the student did not apply his answer to the time interval, which was given in the question.
If we compare the gaze plot of the "Rocket" task solving in fig. 6 and fig. 7, it can be noticed that both participants fixated the text positioned just before the offered options to a lesser extent (in comparison with other texts in the task).The text contains key information about the time interval, which is necessary to be taken into account for the selection of the correct answer.

High school student P27
From the gaze plot of task solving ( fig. 8) a greater number of fixations on all the texts in the task is noticeable at first sight (i.e. on the introductory sentence and on the individual options). Furthermore, it can be seen that the student-novice, unlike student-expers P47, much more often looked at the pictures (including the name of the task, the small text under the image and the text of the note), while paying a relatively large amount of attention to the image of the boy throwing the ball up. From video recording we can observe that the student-novice first carefully read the individual options, wherein he paid only little attention to the illustrative picture. During the second reading, he carefully viewed all the pictures (including the photos of the children). During the third reading he did not proceed systematically. It is clear that he was hesitating between options A and B and at the end of solving the task he fixated the illustrative picture quite often and for long time.
From the student's answer it is possible to say the student has the typical misconception that on the ball thrown up acts still "force of the hand" (Mandíková & Trna, 2011).

Assessment of graphically rich material
Another aim, that this research followed, was to find out how students deal with perceptually rich materials. The submitted tasks contained, in comparison to common test items with simple cartoons, many graphic elements: headings, photos of children, photo of introductory problem situation, a link to image source and the note.
Typically, we met with the two approaches of the students. Some of the students fixated these elements very little, or not at all. It can be concluded from the analysis of task solving that if students did not fixate pictures, then as a rule they chose their answer (correct or incorrect) quickly, i.e. without longer hesitation (see, e.g. fig. 5 by the student-expert P47 and fig. 6 by the student-novice P39).
On further reflection, the students indicated that this is a test, and therefore, they did not want to be distracted by these grafical elements and they wanted to concentrate only on the task text.
The second group of students showed some interest in the graphics to varying degrees. They often fixated the photos of the characters expressing the statement, with which the students identified the most, choosing it as their answer. This is noticeable in case of most of the students who were included in our research. Furthermore, they were often interested in the illustrative photo of the particular task (the boy on a skateboard and the boy throwing up the ball), including the link to the image. However, none of the participants of the research memorized the web address, they only checked, that it is really a so-called link, and then they focused their attention elsewhere. This corresponds to one or two fixations on the text of the link.
Surprisingly, the students did not fixate the illustrative photo of Albert Einstein at all; however, they were very often aware of it, as they said during the subsequent reflection. Apparently, it is such a well-known image, it may be that their less sharp peripheral vision was sufficient to recognize the content of the photo.

Discussion and Conclusion
In our case study, we conducted a detailed analysis of strategies of solving physics tasks for three students-experts and three student-novices, in which we monitored in particular the total time of solving the task. Based on gaze plots and videorecordings of students eye-movements, we described students' allocation of attention to various task elements and transitions among them, especially among the offered options. We focused on the frequency of fixations on words relevant for the correct solution of tasks as well.
For students-experts, a careful but often fairly quick reading of the entire text of the taskfixating the right keywords or phrases and quick elimination of incorrect solutions was typical.
The students-novices were often solving the task for a longer period of time and they devoted more time to browsing misconceptions. This is well-evident for example in solving the task "Ball" by student-novice P27. In contrast, for example, a student-novice P39 was very clearly convinced of her solution, and after a cursory reading of all the available options she quickly chose her answer. When students solved the R-FCI tasks from the test and chose an incorrect answer, in most cases they showed the fewest average fixations duration on the correct option (Kekule et al., 2017). From the gaze plots of task solving and also from the video recordings it is possible to trace that student-novices less often fixate key words or phrases in the text that are relevant for the correct solution of the tasks (see, for example, solving the task "Skateboard" by student-novice P46). These results show previous studies as well (e.g. Madsen et al., 2012;Jouni et al., 2017). Furthermore, students who belonged to the group of "worst performers" tended to fixate more often the same point within the area of the task. Tendency of the worst test performers to fixate the same point within a graph was observed when students were solving several tasks from the Test of Understanding Graphs in Kinematics (Kekule, 2014b).
All of the tasks, the solution of which we have analyzed above, included some of the most common misconceptions in dynamics. From the gaze plots and video recordings of task solving, misconceptions that persist in students'minds can be well identified (see, for example, "force of the hand" in case of student P27). Sometimes, quick choice of incorrect answer may indicate a considerable degree of the deeprootedness of the misconception (see, for example, solving of the "Skateboard" task by student P39). On the other hand, some students show more hesitation when choosing between the options (see, for example, solving the "Skateboard" task student P54).
In the analysis of the solving strategy used by the individual research participants we also monitored the frequency of the fixation of the pictures and other graphic elements that occuredin the tasks. On the basis of the results mentioned in chapt. 2, it can be said that the students probably fixate the images if they are thinking about something, deciding, etc. From the video recordings it is apparent that the students pay greater attention to pictures especially when they read the text of the task repeatedly and also in the case of a longer hesitation over their answer (as it is for example in solving of the task "Ball" by student P27).
Also, photos of the characters often constituted significant perceptual elements, which attracted the attention of the students. As a disruptive perceptual elements, we can identify the link under the illustration picture, the text of the note and and also the colorful clothing of the characters in the photos, which sometimes unnecessarily attracts students' attention.
Although we are aware that the eye tracking research method has its various limitations (Kekule, 2014a), it can be very useful for observation of students' thinking processes. For example, when we administer a test to the students, we get only students' final answer to a task. As we showed in the presented case studies, students' thinking processes are much richer and vary even among the group of the worst or the best performers. These observations and findings may help teachers in their better understanding of students' problem-solving strategies and can facilitate them in the removal of frequent students' misconceptions.