Quantitative and Qualitative Analysis of the Mental Models Deployed by Undergraduate Students in Explaining Thermally Activated Phenomena

In this contribution we describe a research aimed at pointing out the quality of mental models undergraduate engineering students deploy when asked to create explanations for phenomena/processes and/or use a given model in the same context. Student responses to a specially designed written questionnaire are initially analyzed using researcher-generated categories of reasoning, based on the Physics Education Research literature on student understanding of the relevant physics content. The inferred students’ mental models about the analyzed phenomena are categorized as practical, descriptive, or explanatory, based on an analysis of student responses to the questionnaire. A qualitative analysis of interviews conducted with students after the questionnaire administration is also used to deepen some aspects which emerged from the quantitative analysis and validate the results obtained.


Introduction
Among many cognitive theories, those explaining student reasoning in terms of structured cognitive conceptions, or mental models (Johnson-Laird, 1983), are of special interest for physics education.For this reason, many research papers (Bao & Redish, 2006;Maloney & Siegler, 1993;Carley & Palmquist, 1992;Corpuz & Rebello, 2011;Chittleborough & Treagust, 2007) studied students' understanding of models in different contexts, the mental models deployed by students in order to make sense of given phenomenology, and their expressed forms (Gilbert & Boulter, 1998), often using qualitative or quantitative analysis methods.
However, in the last years there has been a move in social science towards multimethod approaches, which tend to emphasize the breadth of information which the use of more than one analysis method may provide to the researcher (Tashakkori & Teddlie, 2003;Brewer & Hunter, 2006).Research results on eliciting and characterizing student mental models, based on the joint use of quantitative and qualitative methods, can be found in the literature (Hrepic, Zollman & Rebello, 2005;Bao, 1999).Our paper develops this research context and is mainly focused on the discussion of students' scientific explanations (Gilbert, Boulter & Rutherford, 1998) to an everyday life phenomenon, relating it to the physics and chemistry they have already studied in previous courses.The focus is on systems for which a process is thermally activated by overcoming a well-defined potential barrier, ΔE, and is therefore described by an equation containing the Boltzmann factor, exp(−ΔE/KT ), where T is the system temperature and K is the Boltzmann constant.
The method involves the construction of a tool (a specially designed open-answer questionnaire) and a quantitative analysis of student responses, supported by the qualitative analysis of specifically designed interviews.The questionnaire items are reported in the Appendix and are better discussed in (Fazio, Battaglia & Di Paola, 2013), where more detail on the whole research are reported and the related results are studied by using a quantitative method based on statistical implicative analysis, different from the one we present here.The study is performed by analyzing the expressed forms of the mental models student use when tackling a written questionnaire and interviews, i.e. their "answering strategies".
The results discussed here have been obtained with students of the 3-year Bachelor Degree Program in Chemical Engineering at the University of Palermo (UniPA), Italy.In the next sections we present the different steps of our research by explaining the research questions, methods and data analysis, and discuss our results.

Research sample
Our research sample consists of 34 freshmen, enrolled in the Chemical Engineering Degree Program during the Academic Year 2010/2011 at UniPA.During the 1st semester of their Degree Program the students attended general mathematics, physics and inorganic chemistry courses, and they had already passed the related exams.When requested to participate in our study, they were attending a 2 nd semester Physics course dealing with the fundamentals of electromagnetism, and voluntarily chose to participate in the survey.The total number of students on the course was about 60.

Research questions
Following the general theoretical framework and the research aims discussed above, this paper directly addresses the following research questions: • What are the characteristics of the mental models students deploy when searching for explanations to phenomena/situations related to real-life and to subjects studied in previously attended courses?• Do students highlight consistency in their deployment of mental models?

Methodology
The general lines used for this research are summarized in six "steps", that are shown below.More detail can be found in (Fazio, Battaglia & Di Paola, 2013).
Step 1: The questionnaire items (reported in the Appendix) are formulated on the basis of a review of Educational Research literature and a survey conducted with some UniPA university teachers.
Step 2: Validation of the questionnaire is performed: 5 physics freshmen, coming from the same secondary schools attended by our student sample, are asked to highlight problems in the questions, like unclear or ambiguous terminology.
Then researchers make an independent analysis of the possible (a-priori ) student responses to the questionnaire items, which results in the singling out of a set of possible answering strategies for each item (Brousseau, 1997).
Step 3: After the submission of the questionnaire to the research sample, researchers independently analyze actual student responses to each item and compare them with the a-priori found answering strategies, adding new ones as needed.The questionnaire items and the related student answering strategies are reported in the Appendix.
Step 4: It is assumed that each student has a latent cognitive structure underlying their answers to the questionnaire items, referred to as a "mental model".
Answering strategies are grouped into idealized sets.Each set is synthesized by typical reasoning procedures that allow us to infer an epistemic category of students' mental models, defined as "practical/everyday", "descriptive", or "explicative".
Step 5: The extent to which actual student answering strategies correspond to the idealized categories is studied by using quantitative analysis methods (Gower, 1966;Mantegna, 1999).
Step 6: An interview protocol is designed by the researchers and interviews are taken with a subset of the student sample in order to extend and validate the results obtained by means of the quantitative analysis.The interviews are conducted immediately after the questionnaire submission, on a voluntary basis.The interview questions are aimed at supplying relevant information about the meaning of students' answers and at widening the analysis of their answering strategies, highlighting points of interest or unusual elements in the questionnaire answers.Checking the validity of the questionnaire items in actually revealing the students' reasoning when constructing explanations was another aim of the interviews.The interview protocol is pre-designed by all three researchers, but the interviews are conducted by one of them, face to face with the students.In many cases, questions not included in the interview protocol are asked, in order to better clarify specific situations which emerged during the discussion.

Questionnaire analysis
During the analysis of the student answering strategies, each researcher draw up a table summarizing them.Discordances between researchers' tables were found in some cases, when a student answer was classified under not just one of the a-priori/aposteriori strategies, but two or more of them.In a few cases, discordances were due to different researchers' interpretations of students' statements.This happened 19 times when comparing the tables of researchers 1 and 2, 17 times for researchers 1 and 3 and 16 times for researchers 2 and 3. Hence, a good inter-rater reliability of the analysis is demonstrated, with accordance percentages of about 91-92 % between the analysis tables of each pair of researchers.The differences between the three tables were compared and discussed by the researchers to reach a consensus on a common table to use for the study.
The careful reading of the students' answers to the questionnaire items, within a framework provided by domain-specific expertise and previous research in the field of the description of student modelling competencies (Sperandeo-Mineo, Fazio & Tarantino, 2006), allowed us to classify students' responses into three phenomenographic (Marton, 1988;Marton & Booth, 1997) categories of mental models.They are Practical/Everyday, Descriptive and Explicative, as described in Table 1, where the reasoning procedures representative of each model category are also shown.The student describes and characterizes the analyzed process by finding/remembering the relevant variables and/or recalling from memory their relations, expressing them by means of different language (verbal, iconic, mathematic).He/she does not explain the causal relations of the physics parameters involved on the basis of a functioning model (microscopic/macroscopic).
The student proposes a model (qualitative and/or quantitative) based on a cause/effect relation or provides an explanatory hypothesis by introducing models which can be seen at a theoretical level.
We then built a table which identifies three 'idealized sets' containing the answering strategies that can be considered typical of each mental model category shown in Table 1.Each set defines the ideal profile of a student answering all the questionnaire items always using strategies related to the same category of mental model.These profiles have been used for a similarity analysis between them and the real students, as explained in the following.More detail can be found in (Fazio, Battaglia & Di Paola, 2013).
In order to study the "similarity" between the students and the three categories of mental models we identified in Step 4 of our analysis, we compared the answers given by the students with the answers typical of each ideal student profile, and calculated the Pearson's correlation coefficients, r ij between each students and the three profiles, where i (i = 1, 2, . . ., 34) denotes a generic student and j represents one of the three ideal student profiles.By following a methodology well known in the field of Econophysics (Mantegna, 1999), where it is common to compare the behavior of real stocks traded in financial markets with the characteristics of "idealtype" stocks, like banking, industrial, service, etc., the "distances" between each student and the three ideal profiles (i.e. the student mental model profiles) were calculated by using the relationship: The general idea behind the use of this definition of distance between two elements i and j is that pairs of elements with positive correlation coefficient are "more similar" than pairs with correlation coefficient zero, or negative.In our case, when a student i never answers the questionnaire items by using strategies typical of a given profile j, r ij = −1 and the related value d ij assumes its maximum value, 1.When the student answering strategies are always be found in the same ideal profile j, r ij = 1 and d ij is 0. We used the values d ij to build a graph that can easily evidence if the three mental model categories really describe the real student behavior and if it is possible to identity clusters of student behavior with respect to the mental models.
Figure 1 shows the graph obtained by using our data, where each ideal student profile is represented as one of the vertex of a Reuleaux triangle, whose distance from any of the other two vertexes (i.e.ideal profiles) is equal to 1 (i.e. the maximum distance between two elements in our analysis).In this graph students are represented by S i (where i again goes from 1 to 34) and are placed within the triangle according to their distances with respect to the three ideal student profiles.From Figure 1 it can be seen that many students are far away from a given profile of less than 0.5.This means that they appear to have answered the questionnaire items by putting into action well defined mental models.However, a number of students is distributed in proximity of the centre of the triangle; this means that their distances with respect to the three profiles are comparable, i.e. these students seem to use a variety of mental models when tackling the questionnaire items.Going into detail, 8 students can be classified as mainly putting into action Everyday-type strategies, 16 highlight the use of Descriptive-type ones (although many of them have distances near, or equal, to 0.5 with respect to this profile), and only four can be considered as mainly using Explicative-type mental models.
The analysis here reported is coherent with a more detailed study of the similarity between the students and the three ideal profiles (Fazio, Battaglia & Di Paola, 2013) performed by using a more complex approach based on Statistical Implicative Analysis (SIA) (Lerman, 1981;Lerman, Gras & Rostam, 1981a, 1981b).

Interview analysis
According to many research papers (Berg, 1989;Onwuegbuzie et al., 2012) a detailed analysis of the language used by each student during an interview, or when carrying out an activity involving human interaction, can provide evidence of the cognitive style(s) used when tackling a given issue or problem.Therefore, the interviews were audio recorded and then analyzed by the three researchers, partly on the basis of a search for 'indicator words/utterances' and specific aspects of students' answers which could help to answer the research questions.The analysis of the semantic properties of the student's language was based on the distinction made by the French psychologist Frederic Pauhlan between the sense and the meaning of a word and considering "the preponderance of the sense of a word over its meaning" (Vygotsky, 1986: p. 244): "'the sense is . . . the sum of all the psychological events aroused in our consciousness by the word.It is a dynamic, fluid, complex whole, which has several zones of unequal stability.Meaning is only one of the zones of sense, the most stable and precise zone.A word acquires its sense from the context in which it appears; in different contexts, it changes its sense." Several methods of analyzing interview excerpts are described in previous research on this subject.One such method involves the use of coding schemes to associate the number of indicator word/phrases that occur with specific forms of reasoning (Weber, 1990;Azmita & Montgomery, 1993).However, we acknowledge that "the nature of language -in which any one grammatical form can be used to fulfill a range of pragmatic functions -renders any coding scheme of dubious value if used separately from a more contextually sensitive . . .type of analysis" (Mercer et al., 2004: p. 372).
For this reason when analyzing the interview excerpts we tried to make sense of the students' use of indicator words/utterances in the specific context of the question itself (Onwuegbuzie et al., 2012;Leech & Onwuegbuzie, 2007), in order to highlight points of interest or controversial behavior in the related questionnaire answers.Furthermore, we also allowed the interviews, and the related qualitative analysis, to be driven by particularly relevant strategies used by students when answering the questionnaire items, and by their implications, as reported in the introductory remarks of each interview.Table 2 shows some examples of key-words and phrases and specific aspects of the students' answers that we used as evidence of the cognitive style(s) student used when tackling the interviewer answers.
Below we report some examples of answers given by our students to the interviewer questions.In them it is possible to recognize some key-words and phrases we identified as descriptors of a given mental model used to tackle the question.
Eleonora: ". . .molecules act each other by means of electric forces. . ." Luca: ". . .temperature is related to molecular movement , i.e. to molecular energy. . ." Fabiana: ". . .as the mathematical formulas are similar, I think that temperature and energy/enthalpy should play the same roles." Matteo: ". . .I now remember that when studying the vapour pressure equilibrium in liquids." Aldo: "I know from my experience that . . .a minimum temperature must be reached in order to light a real life object, like . . .a match, if you strike it." Here, Eleonora and Luca highlight clear references to microscopic models (i.e. the use of explicative-type mental model) in answering the interviewer questions.Fabiana and Matteo highlight a Descriptive-like behaviour, with clear references to the use of mathematical formulas, and to the use of memory of studied subjects, to tackle the questions.Aldo shows to recall a real-life experience (striking a match) to tackle the question, highlighting an approach typical of Everyday-type mental model.The first four students can be found in figure 1 graph as actually being classified as Explicative (Eleonora, student S13, and Luca, student S34) or Descriptive (Fabiana, student S23, and Matteo, student S20).On the other hand, Aldo, student S31, is classified as a Descriptive mental model user in Figure 1, that, we recall it, is built only with data coming from the answers to the questionnaire items.This shows that a more in-depth analysis is needed in order to correctly classify a student in a given category, something that can be easily done with the joint use of qualitative interview analysis and quantitative methods.A more complete analysis of Aldo's answers to the interviewer questions, highlighting his use of mixed-type mental models when tackling with problems//situations, can be found in (Fazio, Battaglia & Di Paola, 2013), where many excerpts from student interviews, better characterizing the use we do in our study of interview analysis, can be found.

Discussion and conclusions
The quantitative and qualitative data analysis reported above allow us to answer the research questions, which regard 1) the characteristics of the mental models students deploy when searching for explanations to phenomena/situations related to real-life and to subjects studied in previously attended courses, and 2) the consistency in students' deployment of mental models.
The similarity analysis allowed us to identify clusters of students whose answering strategies can be completely included into categories related to three different mental models.These categories highlight the reasoning procedures "ran" by students when searching for explanations about phenomena and/or proposed situations.
Many of the students, S i , are plotted in Figure 1 graph with distances less than 0.5 with respect to one of the three profiles, highlighting a consistency in their use of a specific mental model when tackling with the situations proposed in the questionnaire items.On the other hand a significant number of students is distributed in proximity of the centre of Figure 1 Reuleaux triangle; this means that their distances with the three profiles are comparable.So, these students seem to use a variety of mental models when tackling the questionnaire items and highlight a lack of consistency in their deployment of mental models.
The analysis of the interviews allows us to go further and better characterize the student behaviour.Many of them clearly show to have more than one view about the nature and use of explications in science.Often strategies which are inefficient at correctly connecting mathematical modeling to real situations are revealed.Very often, reference to a well known mathematical model seems to stimulate a recalling procedure, i.e. a search in memory for examples that fit in with the formula, without a clear understanding of its physical meaning.Moreover, the analysis of interviews also highlight a significant use of approaches based on common-type knowledge, even in students who generally adopt descriptive strategies.
Our results are consistent with data from the literature (Bao & Redish, 2006;Maloney & Siegler, 1993;Carley & Palmquist, 1992;Corpuz & Rebello, 2011;Chittleborough & Treagust, 2007;Hrepic, Zollman & Rebello, 2005;Bao, 1999) showing that the mental models students deploy in creating explanations can be eclectic, and sometimes contradictory.In fact, many students of our sample use different kinds of reasoning, with particular reference to ones which are inefficient for correctly associating explanations to real situations.A significant presence of everyday or descriptive ideas in student answers is highlighted, in some cases even in students who generally use explicative strategies.
the description of the phenomenon and for the construction of an interpretative model of the phenomenon itself.
1peA The relevant quantities are not identified.1peB The relevant quantities are not identified, but a description/explanation based on common sense is given.1deA The relevant quantities are identified, but they are not used properly to give an explanation.1deB Only temperature is identified as relevant, but the phenomenon is not correctly described.1deC Only temperature is identified as relevant.It is used to give a rough description of the phenomenon.1deD The phenomenon is described by means of the macroscopic variables pressure and volume, but a microscopic model is not identified.1deE The phenomenon is described by means of the macroscopic variables temperature, energy and heat, but a microscopic model is not identified.1deF The phenomenon is described by means of a mathematical formula, but a microscopic model is not identified.1exA The phenomenon is not adequately described (by means of a mathematical formula or verbally), but a microscopic "functioning mechanism" is roughly presented in terms of "molecular collisions".1exB The phenomenon is not adequately described (by means of a mathematical formula or verbally), but a microscopic "functioning mechanism" is presented in terms of energy exchange between molecules.1exC The phenomenon is verbally described and a microscopic "functioning mechanism" is roughly sketched.1exD The phenomenon is described by means of mathematical relations between macroscopic quantities and a microscopic "functioning mechanism" is found.2. In chemical kinetics it is well known that the rate of a reaction, u, between two reactants follows the Arrhenius law: Describe each listed quantity, clarifying its physical meaning and the relations with the other quantities.
2peA The fundamental quantities are not described and/or only examples of its application to everyday-life phenomenology are given.2peB Some quantities are mentioned, but no description of the process is given.2deA The relevant quantities are found, but only a few are described in terms of their physical meaning.2deB The relevant quantities are found, but only described in terms of their mathematical meaning in the formula.No relation between them is identified.2deC The relevant quantities are found and correctly described in terms of their physical meaning.No relation between them is identified.2exA The relevant quantities are found and correctly described in terms of their physical meaning.Some relations between them are identified.
2exB The relevant quantities are found and correctly described in terms of their physical meaning.The relations between them are correctly identified.3. What do you think the role of a catalyst is, in the development of a chemical reaction?
3peA A definition of catalyst is given, which does not conform to the scientifically correct one.3peB A definition of catalyst based on an analogy with the concept of enzyme is given.The analogy is recalled without providing additional reasoning.3deA The catalyst is described as a substance which speeds up a chemical reaction.No additional explanation is supplied.3deB The catalyst is described as a substance which shifts the chemical equilibrium towards the products.No additional explanation is supplied.3deC The catalyst is described as a substance which speeds up a chemical reaction.An explanation is given using common language.3deD The catalyst is presented as a substance which shifts the chemical equilibrium towards the products.An explanation is given using common language.3deE The catalyst is presented as a substance which speeds up a chemical reaction.The concept is generically described in terms of energy.3deF The catalyst is presented as a substance which shifts the chemical equilibrium towards the products.The concept is generically described in terms of energy.3deG The catalyst is presented as a substance which speeds up a chemical reaction.The concept is described by simply citing the energy gap concept, without any explanation.3deH The catalyst is presented as a substance which shifts the chemical equilibrium towards the products.The concept is described by simply citing the energy gap concept, without any explanation.3deI The role of a catalyst in a chemical reaction is discussed referring to the energy gap concept, but only in macroscopic terms.3exA The role of a catalyst in a chemical reaction is discussed taking into account the energy gap concept.The concept is explained considering a microscopic model regarding collisions between molecules.3exB The role of a catalyst in a chemical reaction is discussed taking into account the energy gap concept.The concept is explained considering a microscopic model which links the energy gap concept with the molecular energy.4. Can you give your own microscopic interpretation (model) of the Arrhenius law?
4peA Everyday-life concepts are mentioned, without any correct relation to the Arrhenius law.4deA Scientific concepts, such as energy, temperature or molecular thermal agitation, are mentioned, but they are not correctly related to the Arrhenius law.4deB Arrhenius law is described as a mathematical function of T or E. No explanation of the meaning of these quantities is given.
4deC Arrhenius law is described as a mathematical function of both T and E.
No explanation of the meaning of these quantities is given.4deD Arrhenius law is described as a function of both T and E and the meaning of these two quantities is outlined mainly in mathematical terms.4deE Arrhenius law is described as a function of both T and E. The physical meaning of these two quantities and/or of their ratio in the Arrhenius law is outlined.4deF Arrhenius law is described outlining the physical quantities involved.
Collision theory is sometimes mentioned, but a clear reference to a microscopic model is not always present.4exA A generic explanation based on a microscopic model of collisions between molecules is given.The activation energy concept is outlined but its relation with kT is not clearly presented.4exB A quantitative explanation in terms of the "collision theory" is given.
A correct microscopic model is presented and the role of the activation energy and of kT is clearly expressed.5. Can you think of other natural phenomena which can be explained by a similar model?5peA A few phenomena not related to the model are mentioned.No explanation is given.5peB A few phenomena not related to the model are mentioned.An explanation is given using common language.5deA A few phenomena not related to the model are mentioned.An explanation is given using mathematical formulas.5deB Some phenomena related to the model are mentioned, but these are limited to the context of the attended graduation program (chemical engineering).An explanation is given using mathematical formulas.5deC Some phenomena related to the model are mentioned, and non-chemical phenomena are also taken into account, but a clear explanation is not given.5deD Some phenomena related to the model are mentioned, and non-chemical phenomena are also taken into account.An explanation is given using mathematical formulas.5exA Some phenomena related to the model are mentioned, but these are limited to the context of the attended graduation program (chemical engineering).An explanation is given outlining a common microscopic model.5exB Some phenomena related to the model are mentioned, and non-chemical phenomena are also taken into account.An explanation is given outlining a common microscopic model, but energy and temperature are not clearly interrelated.5exC Some phenomena related to the model are mentioned, and non-chemical phenomena are also taken into account.An explanation is given outlining a common microscopic model.The role of energy and temperature in the model is clearly discussed.6.Which similarities can be identified in the previous phenomena?Is it possible to find a common physical quantity which characterizes all the systems you discussed in the previous questions?
6peA No similarities are detected and questions 1) and 2) are identified as being related to a different context on the basis of everyday-life reasoning.6deA No similarities are detected and questions 1) and 2) are identified as being related to a different context.An explanation is given, mentioning physical quantities which are not really relevant to the correct explanation of the questions.6deB A few correct similarities are found, but physical quantities are given, which are not really relevant to the correct explanation of the questions.6deC Incorrect similarities are found on the basis of a mathematical formula.6deD A few correct similarities are found on the basis of a mathematical formula.6deE Correct similarities are found, but E and T are not always considered common to all phenomena.6deF Some correct similarities are found.E or T is considered to be characteristic of the various phenomena, but a clear justification is not given.6deG Some correct similarities are found.E or T is considered to be characteristic of the various phenomena, clearly explaining why.6deH Some correct similarities are found.E or T is considered to be characteristic of the various phenomena, but the relevance of their ratio in explaining the energy threshold processes is not clearly presented.6exA Some correct similarities are found.E or T is considered to be characteristic of the various phenomena.The activation energy role is correctly discussed in all the mentioned phenomena, but only in macroscopic terms.6exB Some correct similarities are found.E or T is considered to be characteristic of the various phenomena.The activation energy role is correctly discussed in all the mentioned phenomena, on the basis of a microscopic model.

Figure 1 :
Figure 1: Graph of "distances" between real students and the three ideal student profiles

Table 1 :
Categories of mental models deployed by students when tackling the questionnaire and the related reasoning procedures

Table 2 :
Examples of key-words and phrases and specific aspects of the students' answers typical of the three categories of mental model