Assessing physics pre-service teachers’ conceptions about experimental design

1. Introduction

Experimental design, as a pivotal component of scientific research methodology, evolved from statistics and experimental psychology fields at the 19th-20th century.^[1] It outlines a methodical framework for experiment planning, bolstering research reliability, validity, and reproducibility.^[2,3] Fundamentally, it represents a structured approach to investigating phenomena, hypotheses, and causal relationships through controlled observations and manipulations. This approach is pivotal for advancing knowledge and fostering critical thinking in students.^[4,5] Consequently, experimental design has garnered heightened attention as a quintessential scientific practice in labs.^[5,6] Notably, its significance in students’ education is underscored in national science education documents.^[7,8]

Compared to experimental design, scientific inquiry has been particularly emphasized in science education, especially at the primary and secondary school levels.^[9,10] Scientific inquiry refers to the diverse ways in which scientists study the natural world and develop explanations based on evidence derived from their work.^[11] It describes the processes through which scientific knowledge is constructed, including the conventions of its development, acceptance, and application.^[12] In other words, scientific inquiry provides students with a framework to explain the natural world, ensures their understanding is rooted in observations and experimental data, and offers opportunities to explore how the world operates based on empirical evidence.^[13]

In recent years, researchers have increasingly examined the nuanced, context-dependent practices of scientists in their efforts to make sense of the world. This has led to a “practice turn” in science education, with a growing emphasis on scientific practices, as highlighted in the Next Generation Science Standards (NGSS).^[14] However, the essence of inquiry or scientific inquiry remains unchanged: it emphasizes learners engaging with data, using evidence and logic to interpret events or phenomena, often in collaborative and social settings.^[7,11,15]

Experimental design, as a critical component of scientific inquiry, involves carefully balancing features such as “power,” generalizability, various forms of validity, practicality, and cost.^[16] It provides a robust framework to ensure students can produce meaningful and defensible evidence, rather than relying on statistical analysis to correct defects after the fact. This reduces the likelihood of students being solely driven by data, ensuring their understanding is more aligned with the underlying principles and context.^[17] Moreover, experimental design ensures the core practices of scientific inquiry by addressing the interactions between variables, global knowledge, information quality, and other key factors.^[18]

Thus, this study aims to investigate pre-service teachers’ conceptions of experimental design, including but not limited to its role in scientific inquiry, its ability to produce meaningful evidence, and its broader implications.

Existing evidence indicates that teachers’ instructional practices related to experimental inquiry are influenced by their views on experimental design.^[19,20,21] As pre-service teachers represent the next generation of educators, their perceptions of experimental design may shape their future teaching approaches, thereby impacting the development of future talent in the natural sciences, which are fundamentally reliant on inquiry-based learning.^[22]

Unfortunately, there are significant differences in teachers’ views on experimental design across various studies.^[21,23,24] Furthermore, gender differences in perspectives may impact the development of senior high school students’ experimental inquiry skills, due to the disproportionate number of male and female teachers.^[25,26,27] Therefore, this study aims to investigate pre-service physics teachers’ views on experimental design. The findings may contribute to the development of teacher training programs and the curriculum for pre-service teacher instruction.

2. Theoretical Framework

To date, the definition of experimental design remains debated.^[28] Various authorities offered nuanced views on experimental design. The Programme for International Student Assessment (PISA) Scientific Literacy Assessment Framework by the Organization for Economic Co-operation and Development (OECD) offered a foundational perspective, highlighting that it involves identifying, distinguishing viable research questions, proposing methodologies, and ensuring data reliability, and objectivity and generalizability of explanations.^[8] This holistic approach transcends procedural steps, delving into the core of scientific inquiry. Moreover, the National Science Education Standards by the National Research Council detailed it, emphasizing core concepts, equipment use, safety, methodology, technology, idea clarification, and knowledge acquisition.^[7] This aligns with PISA’s emphasis on the comprehensive process of experimental design. The American Association of Physics Teachers (AAPT) advocated student autonomy in posing questions and selecting methods and tools, highlighting human behavior characteristics in experimental design.^[29] The American Association for the Advancement of Science (AAAS) outlined key abilities for experimental design: problem identification, hypothesis-making, planning, and inference, reflecting reflect a deep understanding of experimental design concept based on scientific thinking, emphasizing the rigor and logic underlying the entire process.^[30]

Additionally, some researchers have contributed to the evolving definition of experimental design. Fisher focused on drawing valid conclusions in the presence of nuisance factors, aligning with the effectiveness and quality of experimental design.^[1] This resonates with Deming, Telford and Stavros et al.^[31,32] Hibbert’s characterization as a statistical technique for planning, conducting, analyzing, and interpreting data underscores the statistical foundation of experimental design.^[28] Brownell et al. emphasized experimental design is essential for achieving success in science and gaining fluency in critical thinking, broadening its scope to scientific thinking.^[33] Mistry and Gorman focused on students’ selection of experimental devices, methods, and analysis techniques, underscoring the value of experimental design in fostering hands-on learning and practical application.^[34]

This study aims to assess physics pre-service teachers’ conceptions about experimental design. Considering the existing definitions related to experimental design, six key elements of experimental design identified in previous research are utilized to guide the present study: human behavior characteristics; the comprehensive process of experimental design; the statistical foundations of experimental design; the effectiveness and quality of experimental design; scientific thinking; and the value of experimental design (see Table 1 for specific connotations).

Table 1. Key elements of experimental design and their meanings

Key element	Meaning
Human Behavior Characteristics[29]	The active engagement and autonomy of individuals in posing questions, selecting methods, and interpreting results.
Comprehensive Process[7,8]	A comprehensive process means designing a series of steps required for experimental inquiry, such as defining objectives, identifying problems, planning procedures, conducting experiments, recording data, analyzing results, and drawing conclusions.
Statistical Foundations[28]	The use of statistical techniques for planning, conducting, analyzing, and interpreting data.
Effectiveness and Quality[1,31,32]	The focus on improving the effectiveness and quality of designs, including the management of nuisance factors.
Scientific Thinking [30,33]	The critical thinking skills required for hypothesis formulation, experimental procedure planning, and inference-making through findings interpretation.
The Value of Experimental Design[33,34]	Its fundamental role in achieving success in science and gaining fluency in critical thinking, essential for fostering scientific literacy and inquiry-based learning

Based on the previous definition, synthetically, we hold that experimental design can be defined as a comprehensive, holistic process that involves identifying research questions, proposing methodologies for exploration, ensuring data reliability and generalizability, and drawing valid conclusions based on statistical foundations. It encompasses human behavior characteristics such as autonomy and engagement, focuses on the effectiveness and quality of designs, and cultivates scientific thinking and critical skills essential for success in science.

3. Literature Review

Several studies have explored pre-service teachers’ understanding of experimental design. For instance, Seung et al. used an open questionnaire to investigate elementary pre-service teachers’ understandings of science practice and found that they were already aware of the importance of experimental design in scientific practice.^[35] Zhu et al.examined the impact of inquiry-based design thinking (IBDT) on pre-service teachers’ performance in designing experiments.^[21] They found that most pre-service teachers believed that they have insufficient understanding of experimental design (e.g., its definition, purpose or procedures), consistenting with the findings of Molefe and Aubin, Özer and Sarıbaş that pre-service teachers lack sufficient understanding of experimental design.^[23,24] However, IBDT approach improved their understanding of experiment design and innovative pedagogy for experiment design. Moreover, Yoon et al. found pre-service elementary teachers’ challenges in guiding children to devise effective experiments for their hypotheses, emphasizing the need for teacher educators to elevate pre-service teachers’ cognitive understanding and skills.^[20] Hacıeminoğlu et al. uncovered that pre-service teachers exhibit errors in proposing hypotheses or identifying variables during the experimental design process.^[36] Additionally, Zulfiani and Herlanti discovered no gender difference in experimental design among pre-service teachers, aligning with the perspective of Sieberg, who observed no significant gender differences in experimental design levels between male and female students and suggested that teaching technology is equally advantageous or disadvantageous to both genders.^[37,38] Nevertheless, this contradicts the findings of Xiantong et al., who reported the existence of gender differences in students’ experimental design level.^[39]

Existing studies provide several significant conclusions. Firstly, there are notable differences in pre-service teachers’ views on experimental design across various studies, reflecting varying levels of comprehension.^[21,23,24] This imbalance in understanding of the concept of experimental design may lead to contradictory instructional practices in their future teaching and could result in insufficient experimental inquiry skills among their future students.^[21,40,41] To date, there has been a lack of research measuring pre-service teachers’ conceptions about experimental design from a holistic perspective (i.e., considering all aspects of experimental design). Moreover, in China, there is even a lack of specific research assessing pre-service teachers’ views on some aspects of experimental design. Therefore, considering these issues, it is necessary to investigate the views of pre-service physics teachers on experimental design.

Moreover, evidence indicates that individual pre-service teachers focus on different dimensions of experimental design to varying extents.^[21,36] For example, some individuals may prioritize certain aspects of experimental design, such as experimental purpose and procedure, over others, which can potentially impact their instructional practices related to experimental inquiry.^[21] These two conclusions partly attribute to the fact that pre-service teachers have their own naive views on experimental design. Additionally, it may also be due to the pre-established theoretical frameworks used by researchers being incomplete or unable to cover all dimensions of experimental design.

Existing studies have also found controversial conclusions regarding gender differences in pre-service teachers’ views on experimental design.^[37,38,39] These findings suggest that males and females might adopt different instructional paths related to experimental inquiry in their future practice.^[37,42] Therefore, this study also explores the views of male and female science pre-service teachers on experimental design separately.

In conclusion, the research questions to be addressed in this study are listed as follows:

1. What views of experimental design do pre-service teachers hold?

2. What differences exist in the focus of attention in the views of experimental design between male and female pre-service teachers?

4. Method

4.1. Participants

This study targeted 90 pre-service middle school science teachers (39 male and 51 female teachers) from two classes in a normal university. These pre-service science teachers will engage in K-12 education after graduation. The participants were junior students, and before participating in this survey, they had all taken Mechanics, Thermology, Physics Education and so on, acquiring the basic knowledge and skills to design and conduct experimental inquiry.

The questionnaire survey was conducted in a supervised classroom environment with one class as a unit, and all pre-service teachers participated voluntarily. Participants was expected to complete the questionnaire within one hour. Prior to survey, they were notified that the survey solely sought their perspectives on experimental design issues and that their responses held no affecting on their school evaluation. Participants were encouraged to express their opinions comprehensively, supported by specific examples.

4.2. Research Design

Given the lack of existing interview protocols for collecting pre-service teachers’ responses about experimental design concept, our study developed its own protocol grounded in a theoretical framework comprising six key elements summarized from previous definitions of experimental design in the literature (detailed in Theoretical Framework section).

Drawing on the questionnaire design method used by Al Sultan et al. to investigate perceptions of a certain concept, our protocol first included an open-ended question (Have you heard of the term ‘experimental design’? Tell me what this term means from your point of view.), and this main question included nine sub-questions that formed the core of the protocol.^[43] Regarding the nine sub-questions, a specific example is provided to illustrate how to design. A key element of previous experimental design definitions concerns human behavior characteristics. Combining the theoretical framework with the characteristics of experimental design, we have designed a sub-question as follows:“Could you describe the distinct behaviors you believe a person proficient in experimental design would exhibit while conducting physics experiments?” This question was designed to probe pre-service teachers’ perceptions of observable behavior exhibited by proficient experimental designers, thereby evaluating their understanding of how human factors influence experimental conduct and outcomes. The remaining sub-questions are also designed following the same procedure as above.

It is necessary to ascertain the validity of the protocol. The protocol underwent review, pilot testing, and was eventually implemented among participants. Specifically, two science education researchers independently reviewed the questionnaire and identified the dimensions associated with each question. They subsequently arranged a meeting to discuss their observations, during which each researcher presented their perspectives on each question and the dimensions it examined. They then compared their views and discussed any discrepancies to reach a consensus on the dimensions evaluated for each question. Finally, they engaged in further discussion and negotiation regarding the dimensions related to each question to ensure that the survey adequately encompasses the six key elements of theoretical framework. Moreover, researchers randomly selected two science education undergraduates for testing to confirm clarity and brevity of the questions, as well as to address the purpose of the protocol without confusion or ambiguity. Subsequently, the questionnaire survey was conducted among the participants of this study.

Based on the aforementioned content, the present study presents the flowchart of the research design, as illustrated in Figure 1.

4.3. Data Analysis Procedure

Researchers employed an inductive process for open coding through the tool of MAXQDA. Through iterative reviews, they developed a codebook tailored to the objectives and questions of the current research. The first and second researchers independently coded and reviewed all texts, linking them to the definition of experimental design. As new codes emerged, researchers discussed and harmonized them until consensus was reached. The inductive analysis generated a list of codes, which were then merged into related categories and interpretive themes through an iterative process. Through extensive discussions, researchers made a deductive comparison of the interpretive themes with the components of the experimental design.

4.4. Statistical Analysis

Upon completion of coding, MAXQDA provides valuable data that facilitate the discussion of relevant issues. On one hand, MAXQDA provides the number of codes for each theme across all responses, as well as the overall number of codes across all responses. This helps in calculating the proportion of codes for a single theme within the total number of codes, indicating the relative proportion of this theme among all themes. On the other hand, MAXQDA provides the specific number of times each theme is coded in each response. It quantifies the number of codes for each theme within different datasets (responses from female pre-service teachers, responses from male pre-service teachers) and calculates the proportion of each theme’s number of codes within the total number of codes in each dataset. This information is helpful in conducting gender-based differential tests for each theme.

5. Results

5.1. Pre-Service Teachers’ Conceptions About Experimental Design

By analyzing the responses of pre-service physics teachers to the questionnaire, the following ten themes were identified, highlighting that their conceptions about experimental design primarily center around the following ten aspects: Advancing Scientific Research through Design; Clear and Complete Experimental Procedure; Evidence-Based Nature of Experimental Data; Theoretical and Instrumental Basis for Experimental Research; Problem-Centered Research Approach; Purpose-Driven Experimental Design; Real-World Relevance of Research Motivations; The Centrality of Experimental Design in Science Teaching; The Inquiry-Based Design Thinking; and The Iterative and Self-Correcting Nature of Scientific Inquiry. Table 2 provides the relevant meanings of these ten themes and the examples quoted by participants in their responses.

Figure 2 presents the proportions of all themes. In this figure, the horizontal axis represents the proportion of theme, while the vertical axis represents the themes themselves. It is obviously that the theme “Evidence-Based Nature of Experimental Data” accounts for approximately 17.7% of all coding, making it the most prevalent one. Assessing topic proportions can indicate where pre-service teachers’ attention is focused, and the results suggest that their focus on experimental design is unbalanced among different themes.

Figure 2 illustrates theme “Evidence-Based Nature of Experimental Data” accounts for the largest proportion among ten themes, suggesting finding evidence based on data to solve problems is pre-service teachers’ focus of attention. Participants’ response scripts also support this, such as emphasizing the acquisition and analysis of data is vital for obtaining research evidence, and logical experimental conclusions are intimately tied to rigorous data analysis.

Furthermore, as illustrated in Figure 2, pre-service teachers exhibited varying levels of

Table 2. The meanings of themes and the examples quoted by participants in their responses

Theme	Theme Name	Definition	Quotes from responses
Theme 1	Advancing Scientific Research through Design	Experimental design is viewed as a key driver of scientific progress, fostering innovation and the development of new knowledge and technologies.	Inquiry process design advances scientific research, guiding its direction and offering ideological guidance. A good inquiry process design provides a way to discover and explore new phenomena, enabling scientists to observe and measure unknown phenomena under controlled conditions, deepening the understanding of natural laws and birthing new scientific theories.
Theme 2	Clear and Complete Experimental Procedure	Experimental procedure refers to a detailed and comprehensive set of instructions that outlines every step of an experiment, from preparation to conclusion.	Students can design a reasonable experiment plan to guide the experiment by thinking and analyzing the problem. In the process of experimental design, students should design a complete procedure including inquiry questions, experimental methods, experimental operation procedures, data collection and processing, and the summary of rules.
Theme 3	Evidence-Based Nature of Experimental Data	Experimental data is seen as the crucial evidentiary source for addressing research problems, highlighting its importance in scientific inquiry.	Data is the foundation for testing hypotheses and drawing inferences, offering crucial evidence for objectively evaluating experimental results. Data is pivotal in supporting or refuting tentative hypotheses during the experimental design. Factual data is the most powerful evidence for justifying a hypothesis, while mathematical relationships derived from data enhance its credibility.
Theme 4	Theoretical and Instrumental Basis for Experimental Research	The theoretical knowledge foundation of the experiments, along with the understanding and application of experimental instruments, is considered a critical prerequisite for conducting experimental research. Their essential role in data collection and analysis is particularly emphasized.	The design of physical experiments cannot be separated from the support of physical theories. Only with sufficient knowledge reserves can one have the ability to correctly analyze data. The important step of the experiment is the selection of instruments, necessitating thorough consideration of all potential instruments. Ensuring the integrity of experimental instruments, conducting pre-experiment checks, and promptly replacing faulty equipment are crucial for conducting experiments and collecting accurate data.
Theme 5	Problem-Centered Research Approach	The research problem is identified as the starting point of scientific research, guiding the direction and focus of experimental design.	An accurate, inquirable question is the foundation for conducting experiments, guiding all subsequent inquiries. The genesis of an experiment lies in posing a question. The entire experiment inquiry revolves around the guiding questions.
Theme 6	Purpose-Driven Experimental Design	The research purpose is recognized as the guiding star of experimental design, shaping the overall research trajectory and ensuring that the experiments are aligned with the broader goals and objectives of the research.	The entire experimental inquiry process revolves around research purpose. With a clear purpose, one can analyse and design the experiment's core content, grasp the research object, and define physics variables. A defined goal ensures a focused direction.
Theme 7	Real-World Relevance of Research Motivations	Teachers introduce research motivations that are grounded in real-life scenarios or students' misconceptions, emphasizing the practical applications and relevance of their research work.	Integrate real-life scenarios into the classroom to enhance students' connection between experiments and daily life. Use life situations encompassing experiment principles to introduce experiments, arousing students' interest. Observe diverse life situations, gather students' misconceptions, and design experimental contexts accordingly.
Theme 8	The Centrality of Experimental Design in Science Teaching	Experimental design is recognized as a fundamental skill for science teachers, enabling them to successfully create experiments and effectively implement inquiry-based teaching methods, thereby fostering a deeper understanding of scientific concepts.	Experimental design refers to teachers' ability to design physics experiments for teaching, craft necessary demonstrations, and practical links, and seamlessly integrate them into the classroom. Pre-service teachers can enhance their experimental teaching abilities by writing experimental design reports, ensuring smooth experimental teaching in the future. Teachers should have the ability to design pertinent experiments to foster students' understanding and mastery of relevant knowledge.
Theme 9	The Inquiry-Based Design Thinking	Experimental design thinking as a kind of scientific thinking refers to the comprehensive consideration before implementing the experiment.	Verify the rationality and reliability of the design through scientific thinking. Scientific thinking makes experimental design thinking rigorous and logical, and the steps are detailed. Scientific thinking runs through the whole experimental design, leading the overall experimental idea.
Theme 10	The Iterative and Self-Correcting Nature of Scientific Inquiry	Repeated experimentation is seen as an essential aspect of scientific inquiry, enabling researchers to identify and minimize experimental errors, refine their methods, and ultimately achieve greater accuracy and precision in their findings.	Multiple experiments enhances reliability by deriving consistent results. Repeating experiments can mitigate accidental errors. Ensuring scientific (physics) experimental conclusions' reliability and validity requires ample data, including repeated tests.

attention to different aspects of experimental design. Some content, such as “Evidence-Based Nature of Experimental Data,” may be frequently mentioned in the responses of individual participants, whereas other aspects, like “The Centrality of Experimental Design in Science Teaching,” may not be mentioned at all. Consequently, pre-service teachers have not yet formed a comprehensive view of experimental design and display uneven levels of understanding across its different dimensions. This aligns with existing research findings, indicating that pre-service teachers are still in a developmental phase and have not yet established a comprehensive understanding, with disparities in their levels of comprehension across various aspects.^[44]

5.2. Focus of Attention on the Conceptions About Experimental Design Among Male and Female Pre-Service Teachers

This study focuses on the differences in theme prevalence based on gender. Using the information provided by MAXQDA, this study conducted gender-based difference tests for each theme, with the results presented in Table 3. The result shows that theme “Evidence-Based Nature of Experimental Data” and theme “Problem-Centered Research Approach” reach the statistical significant level (p <.05), indicating that the gender variable may have influenced physics pre-service teachers to spend different time discussing these two themes.

Table 3. Prevalence of topic based on gender

Theme	t statistic	p value
Advancing Scientific Research through Design	0.2791	0.7808
Clear and Complete Experimental Procedure	-0.4374	0.6630
Evidence-Based Nature of Experimental Data	2.5333	0.0134
Theoretical and Instrumental Basis for Experimental Research	-0.6104	0.5437
Problem-Centered Research Approach	2.3183	0.0228
Purpose-Driven Experimental Design	0.6104	0.5432
Real-World Relevance of Research Motivations	1.8779	0.0637
The Centrality of Experimental Design in Science Teaching	-0.1442	0.8857
The Inquiry-Based Design Thinking	1.1040	0.2727
The Iterative and Self-Correcting Nature of Scientific Inquiry	0.4438	0.6584

Gender differences in pre-service teachers’ views on experimental design include both the prevalence of each topic from a quantitative perspective and the focus of content from a qualitative perspective.

Table 3 indicates that the estimates for themes “Evidence-Based Nature of Experimental Data” and “Problem-Centered Research Approach” are statistically significant (p <.05), suggesting gender-differentiated prevalence in these two topics. This means that these topics might attract different levels of interest from male and female pre-service teachers.

However, the themes mentioned in the previous section regarding pre-service teachers’ conceptions about experimental design cannot explain gender-based differences in emphasis. Fortunately, the coding process using MAXQDA allows us to clearly observe the coding content of male or female pre-service teachers on each theme, helping us understand male and female attention within specific topics.

Given the space constraints, we take theme “Evidence-Based Nature of Experimental Data” as an example to explain the gender-based difference of focus contents. Data analysis (including data collection, modelling, and result interpretation) is pivotal in inquiry process, ensuring effectiveness of evidences, reliability and credibility of conclusions.^[45,46] In the responses of female pre-service teachers on this theme, they frequently use terms such as data, evidence, collect, record, analyse, draw, and verify to express their views. In these terms, terms ‘collect’, ‘record’, and ‘draw’ mean an active engagement in the data gathering process and a willingness to extract insights from the data; terms ‘data’, ‘evidence’, ‘analyse’, and ‘verify’ indicate obtaining relevant evidence based on data requires a rigorous process of data analysis, followed by verification of the obtained evidence to ensure its reliability and validity. Combined with the focus and the meaning of this theme, we infer that female pre-service teachers might tend to toward a process-oriented view, focusing more on data rigor and interpretation. The following example from female pre-service teachers’ responses on this theme confirms our interpretation:‘Data is the basis for testing hypotheses and inferences. By collecting, recording, and analyzing data, hypotheses can be tested and conclusions can be drawn’.

Data analysis outcomes are crucial for providing empirical basis for research findings, ensuring reliability and validity of conclusions, and addressing research problems through experimental inquiry design.^[46] In the responses of male pre-service teachers on the theme “Evidence-Based Nature of Experimental Data”, they frequently use terms such as reliable, obtain, experimental conclusion, correct, and experimental result. In these phrases, terms ‘reliable’ and ‘correct’ emphasize the importance of accuracy and precision in experimental outcomes, while ‘obtain’ implies a more direct, perhaps instrumental, approach to acquiring data. Phrases ‘experimental conclusion’ and ‘experimental result’ highlight a focus on the final products of the experimental process, suggesting a possible tendency to prioritize the conclusion over the intricacies of data analysis. Combined with the focus and the meaning of this theme, we infer that male pre-service teachers might have a more outcome-oriented view, placing more emphasis on the correctness and reliability of the outcomes themselves. The following example from male pre-service teachers’ responses on this theme confirms our interpretation:‘High-quality and reliable data facilitate the acquisition of objective evidence, leading to accurate and trustworthy research findings’.

Following same methodology, gender-based differences in focus contents of theme “Problem-Centered Research Approach” are identified. In this theme, female pre-service teachers emphasize the need to focus on and determine research problem, showing sensitivity to problem identification and formulation, while males tend to engage more directly with problem formulation through posing hypotheses.

6. Discussion and Conclusions

This study first summarized the conceptual definitions of experimental design featured in existing research, deriving six key elements to serve as the theoretical framework for this research. Based on this framework, a questionnaire was developed to investigate pre-service teachers’ conceptions of experimental design. To avoid being constrained by the framework, open coding was applied to participants’ responses, summarizing and synthesizing their perspectives. This approach not only ensures the validity of the research but also enables the discovery of relatively novel insights.^[47] As a result, ten topics emerged, which may seem inconsistent with our predefined framework. Therefore, we compared the summarized results with the predefined framework to ensure a comprehensive analysis, as detailed in Table 4.

Table 4. Comparison of this study’s results and the theoretical framework behind the questionnaire

Key element of theoretical framework	The themes of this study
Human Behavior Characteristics	Theme 7[21]b)
Comprehensive Process	Theme 2[7,8,21]
Effectiveness and Quality	Theme 3[1,8,28,30,31,32], Theme 4[7,29], Theme 9[1,31,32], Theme 10[8]
The Value of Experimental Design	Theme 1[8,33,25], Theme 8[21]
Statistical Foundations	-
Scientific thinking	-
-a)	Theme 5
-	Theme 6

^a)“-”represents that a specific aspect within the theoretical framework has no corresponding theme among the themes derived from this study, or a particular theme from the study lacks a relevant aspect in the theoretical framework; ^b) The references following the theme indicate that these studies involve this theme.

First, all themes identified from participants’ responses are supported by existing studies. In other words, researchers have previously mentioned one or more of these themes in their studies, though no single study has mentioned all the themes. Second, each predefined key aspect of experimental design can be described by more than one pre-service teacher’s ideas. This means that while the framework encompasses all aspects of experimental design, it fails to capture the specific ideas of participants.

As mentioned in the section, experimental design provides a robust framework to ensure students can produce meaningful and defensible evidence, rather than relying on statistical analysis to correct defects after the fact. Referring back to the focus on the previous ten themes investigated in the results section, it is evident that there are some dissimilarities between the scope of pre-service teachers’ conceptions about experimental design and our predefined frameworks. These conceptions not only extend beyond the initial definition of experimental design and our predefined aspects, but also reveal omissions in some specific aspects.

Theme 1 emphasizes the evaluation and role of experimental design, while Theme 8 describes the role of experimental design in inquiry learning, enhancing students’ conceptual understanding of scientific concepts. Both themes align with the sixth aspect of our predefined framework. It should be noted that ideas in Theme 8 might be inconsistent with the goals of experimental design. Consequently, we can conclude that pre-service teachers might have confusion between inquiry learning (or instruction) and experimental design.

Theme 2 focuses on the procedures of experiments provided by experimental design, which is consistent with the second aspect of our predefined framework, the comprehensive process.

Themes 9 and 10 emphasize design attributes, with iteration being a core element. The focus of these two themes guarantees the quality of the outputs of experiments. Theme 3 illustrates the nature of evidence, particularly its role in overturning or accepting hypotheses. Theme 4 clarifies the importance of scientific concepts, principles, and instruments in producing effective data and data analysis. These four themes align with the fourth aspect of our predefined framework.

Theme 7 emphasizes the source of motivation, which can enhance an individual’s engagement in the experimental process and is partially consistent with the first aspect of our predefined framework.

Themes 5 and 6 illustrate the two starting points of experimental design: research goals and research questions. Although different, with the latter being the logical starting point of the former, these two themes are not covered by our predefined framework.^[48]

Additionally, it should be noted that in pre-service teachers’ responses, statistical foundations and scientific thinking were not mentioned. This might indicate that pre-service teachers have not realized the importance, role, and specific contents of these aspects in experimental design. This might suggest potential improvements for pre-service physics teachers’ curricula in the near future.

In sum, the conclusions have drawn in this study are listed as follows:

1. Ten themes were identified regarding pre-service physics teachers’ conceptions about experimental design, such as “Advancing Scientific Research through Design” and “Clear and Complete Experimental Procedure,” and they showed a strong interest in the theme of “Evidence-Based Nature of Experimental Data,” yet some lacked a comprehensive understanding of experimental design, displaying uneven degrees of comprehension on different dimensions of experimental design.

2. Among the ten themes of pre-service teachers’ conceptions about experimental design, there are gender-differentiated attention to two themes “Evidence-Based Nature of Experimental Data” and “Problem-Centered Research Approach.” Female teachers focused more on the process, mainly on the rigor and interpretation of data, and problem formulation process, while males were more outcome-oriented, focusing more on outcome reliability and accuracy.

3. This study reveals both alignment and discrepancies with a predefined theoretical framework in pre-service teachers’ conceptions of experimental design, highlighting the need for improvements in teacher training curricula.

Though this study’s conclusions provide valuable evidence for formulating training programmes for pre-service teachers aimed at increasing their competence in experimental design and teaching. There are even shortcomings in several aspects. Firstly, the sample size was relatively limited. We only surveyed 90 pre-service teachers from a university. This restricted the generalizability of our findings. A more extensive and diverse sample, covering different regions, educational backgrounds, and levels of prior exposure to experimental design, would be needed to draw more universal conclusions. Secondly, the study was cross-sectional in nature. We captured the pre-service teachers’ conceptions at a single point in time. Future research should consider longitudinal studies to better understand how their conceptions of experimental design evolve over the course of their teacher training and early teaching careers.

Conflicts of Interest

The authors declare no conflict of interest.

Funding

This research received no external funding.

Author contributions

Xiaoyu Liu: First author, Formal analysis, Investigation, Software, Visualization, Writing-Original Draft.

Weiling Gui: Second author, Project administration, Supervision, Validation.

Shuaishuai Mi: Third author and Corresponding author, Conceptualization, Methodology, Writing-Review.

References

1. R. A. Fisher. “Design of experiments.” British Medical Journal. 1935, 1, 3923, 554.

2. S. Tian, J. Zhu, C. Chen, Z. Wang. “An assessment scale for evaluating the experimental design ability of elementary science teachers based on primary trait analysis.” Journal of Contemporary Educational Research. 2022, 6, 3, 12.

3. X. Yang, M. Zhang, Y. Zhao, Q. Wang, J. C. Hong. “Relationship between creative thinking and experimental design thinking in science education: Independent or related.” Thinking Skills and Creativity. 2022, 46, 101183.

4. D. Coil, M. P. Wenderoth, M. Cunningham, C. Dirks. “Teaching the process of science: faculty perceptions and an effective methodology.” CBE-Life Sciences Education. 2010, 9, 4, 524.

5. E. Etkina, A. Karelina, M. Ruibal-Villasenor, D. Rosengrant, R. Jordan, C. E. Hmelo-Silver. “Design and reflection help students develop scientific abilities: Learning in introductory physics laboratories.” The Journal of the Learning Sciences. 2010, 19, 1, 54.

6. A. Karelina, E. Etkina. “Acting like a physicist: Student approach study to experimental design.” Physical Review Special Topics-Physics Education Research. 2007, 3, 2, 020106.

7. National Research Council (NRC), National science education standards, National Academies Press, Washington, DC 1996.

8. Organisation for Economic Co-operation and Development (OCED), Pisa 2015 assessment and analytical framework: Science, reading, mathematic, financial literacy and collaborative problem solving, OECD Publishing Paris 2017.

9. W. W. Cobern, D. Schuster, B. Adams, B. Applegate, B. Skjold, A. Undreiu, C. C. Loving, J. D. Gobert. “Experimental comparison of inquiry and direct instruction in science.” Research in Science & Technological Education. 2010, 28, 1, 81.

10. M. Pedaste, A. Baucal, E. Reisenbuk. “Towards a science inquiry test in primary education: development of items and scales.” In ternational Journal of STEM Education. 2021, 8, 1.

11. National Research Council (NRC), Inquiry and the National science education standards, National Academic Press , Washington, DC 2000.

12. S. Schwartz, N. G. Lederman, J. S. Lederman. “An instrument to assess views of scientific inquiry: The vosi questionnaire.” 2008.

13. R. W. Bybee, Scientific inquiry and science teaching. In Scientific inquiry and nature of science: Implications for teaching, learning, and teacher education, Springer Netherlands, Dordrecht 2006.

14. National Research Council (NRC), Next generation science standards: For states, by states, National Academies Press, 2013.

15. B. A. Crawford, From inquiry to scientific practices in the science classroom. In Handbook of research on science education, Routledge, 2014.

16. H. J. Seltman. “Experimental design and analysis.” 2012.

17. F. Espinoza, D. Quarless. “An inquiry-based contextual approach as the primary mode of learning science with microcomputer-based laboratory technology.” Journal of Educational Technology Systems. 2010, 38, 4, 407.

18. R. Leardi. “Experimental design in chemistry: A tutorial.” Analytica chimica acta. 2009, 652, 1-2, 161.

19. X. Yang, Y. Wang. “The relation between teaching self-efficacy and behavior of experimental design teaching in chinese science teachers.” Research in Science & Technological Education. 2023, 1.

20. H. G. Yoon, Y. J. Joung, M. Kim. “The challenges of science inquiry teaching for pre-service teachers in elementary classrooms: Difficulties on and under the scene.” Research in Science Education. 2012, 42, 589.

21. L. Zhu, D. Sun, M. Luo, W. Liu, S. Xue. “Investigating pre-service science teachers’ design performance in laboratory class: The inquiry-based design thinking approach.” Journal of science education and technology. 2024, 33, 1, 30.

22. M. Lucero, M. Valcke, T. Schellens. “Teachers’ beliefs and self-reported use of inquiry in science education in public primary schools.” International journal of science education. 2013, 35, 8, 1407.

23. L. Molefe, J. B. Aubin. “Exploring how science process skills blend with the scientific process: Pre-service teachers’ views following fieldwork experience.” South African Journal of Education. 2021, 41, 2, 1.

24. F. Özer, D. Sarıbaş. “Exploring pre-service science teachers’ understanding of scientific inquiry and scientific practices through a laboratory course.” Science & Education. 2023, 32, 3, 787.

25. D. Evans. “Men still shun teaching as figures show gender imbalance persists.” The times Educational Supplement Cymru. 2011, 361, 5.

26. G. Lassibille, M. L. Navarro Gómez. “Teachers’ job satisfaction and gender imbalance at school.” Education Economics. 2020, 28, 6, 567.

27. K. Zumwalt, E. Craig. “Teachers’ characteristics: Research on the demographic profile.” Studying teacher education: The report of the AERA panel on research and teacher education. 2005, 111.

28. D. B. Hibbert. “Experimental design in chromatography: a tutorial review.” Journal of chromatography B. 2012, 910, 2.

29. American Association of Physics Teachers (AAPT). “Goals of the introductory physics laboratory.” American Journal of Physics. 1998, 66, 6, 483.

30. American Association for the Advancement of Sciencee (AAAS), Vision and change: A call to action, AAAS, Washington, DC 2010.

31. S. N. Politis, P. Colombo, G. Colombo, D. M. Rekkas. “Design of experiments (doe) in pharmaceutical development.” Drug development and industrial pharmacy. 2017, 43, 6, 889.

32. J. K. Telford. “A brief introduction to design of experiments.” Johns Hopkins apl technical digest. 2007, 27, 3, 224.

33. S. E. Brownell, M. P. Wenderoth, R. Theobald, N. Okoroafor, M. Koval, S. Freeman, C. L. Walcher-Chevillet, A. J. Crowe. “How students think about experimental design: novel conceptions revealed by in-class activities.” BioScience. 2014, 64, 2, 125.

34. N. Mistry, S. G. Gorman. “What laboratory skills do students think they possess at the start of university?” Chemistry Education Research and Practice. 2020, 21, 3, 823.

35. E. Seung, S. Park, V. Kite, A. Choi. “Elementary preservice teachers’ understandings and task values of the science practices advocated in the ngss in the us.” Education Sciences. 2023, 13, 4, 371.

36. E. Hacıeminoğlu, N. G. Yıldız, R. Şeker. “Factors related to cognitive reasoning of pre-service teachers’ science process skills: role of experiments at home on meaningful learning.” Sustainability. 2022, 14, 13, 7703.

37. Z. Zulfiani, Y. Herlanti. “Scientific inquiry perception and ability of pre- service teachers.” Journal of Turkish Science Education. 2018, 15, 1, 128.

38. J. L. Sieberg, Measuring experimental design ability: a test to probe critical thinking (Unpublished master’s thesis), Bowling Green State University, 2008.

39. Y. Xiantong, Z. Mengmeng, S. Xin, H. Lan, W. Qiang. “Regional educational equity: A survey on the ability to design scientific experiments of sixth-grade students.” Journal of Baltic Science Education. 2019, 18, 6, 971.

40. K. Dorji, P. Tshering. “Examining Bhutanese Science Teachers’ Epistemic Views of Scientific Inquiry.” Science & Education, 2024, 1.

41. G. Mesci, S. Çavuş-Güngören, F. Yesildag-Hasancebi. “Investigating the development of pre-service science teachers’ nosi views and related teaching practices.” International Journal of Science Education. 2020, 42, 1, 50.

42. J. L. Hosp, S. P. Ardoin. “Assessment for instructional planning.” Assessment for effective intervention. 2008, 33, 2, 69.

43. A. Al Sultan, H. Henson Jr, D. Lickteig. “Assessing preservice elementary teachers’ conceptual understanding of scientific literacy.” Teaching and Teacher Education. 2021, 102, 103327.

44. H. B. Can, Y. Boz. “Development of pre-service teachers’ pedagogical content knowledge and the factors affecting that development: a longitudinal study.” Chemistry Education Research and Practice. 2022, 23, 4, 980.

45. A. P. Dasgupta, T. R. Anderson, N. Pelaez. “Development and validation of a rubric for diagnosing students’ experimental design knowledge and difficulties.” CBE-Life Sciences Education. 2014, 13, 2, 265.

46. G. Grolemund, H. Wickham. “A cognitive interpretation of data analysis.” International Statistical Review. 2014, 82, 2, 184.

47. G. C. Banks, H. M. Woznyj, R. S. Wesslen, R. L. Ross. “A review of best practice recommendations for text analysis in r (and a user-friendly app).” Journal of Business and Psychology. 2018, 33, 445.

48. M. van Hooijdonk, T. Mainhard, E. H. Kroesbergen, J. van Tartwijk. “Creative problem solving in primary education: Exploring the role of fact finding, problem finding, and solution finding across tasks.” Thinking Skills and Creativity. 2020, 37, 100665.