Using Measurement Invariance to Explore the Source of Variation in Basic Medical Science Students’ Evaluation of Teaching Effectiveness

Mahmoud  Alquraan; Sulaf  Alazzam; Hakam  Alkhateeb

doi:10.6000/1929-6029.2023.12.23

Authors

Mahmoud Alquraan Al Ain University, Al Ain, UAE and College of Education, Yarmouk University, Irbid, Jordan https://orcid.org/0000-0002-4682-5003
Sulaf Alazzam College of Education, Yarmouk University, Irbid, Jordan
Hakam Alkhateeb College of Medicine, Yarmouk University, Irbid, Jordan

DOI:

https://doi.org/10.6000/1929-6029.2023.12.23

Keywords:

Basic science medical students, student evaluation of teaching (SET), Measurement Invariance (MI), Health Education, Medicine Program

Abstract

Introduction: Many research studies have shown that students' evaluations of teaching(SET) are affected by different variables without testing the requirement of fair comparisons. These studies have not tested the measurement equivalency of SET surveys according to these variables. Measurement equivalency of SET refers to whether a SET survey is interpreted similarly across different groups of individuals (Variable Levels). Without evidence of measurement invariance across different variables under investigation, the SET ratings should not be compared across these variables and this is the goal of this study.

Methods: Measurement Invariance analysis of SET survey was investigated using 1649 responses to SET of four different medical core courses offered by the College of Science and College of Medicine and from different levels.

Results: The results showed the existence of teaching practices in the SET survey that are not equivalently loaded on its factor across the levels of targeted variables, and the college offered medical courses were a source of variation in basic medical science students’ evaluation of teaching effectiveness. On the other hand, teaching practices in the SET survey are equivalently loaded on its factor across course levels.

Discussion: The study results showed that the SET of medical courses is comparable to the courses only taught by the College of Medicine. These results provide evidence that medical courses are different from other courses offered by other colleges. This means that comparing SET of the College of Medicine with other colleges and colleges of medicine needs to compare SET results at the college level only.

References

Hande H, Kamath S, D’Souza J. Students’ perception of effective teaching practices in a medical school. Education in Medicine Journal 2014; 6(3): 63-66. https://doi.org/10.5959/eimj.v6i3.247 DOI: https://doi.org/10.5959/eimj.v6i3.247

Boerebach B. Evaluating clinicians’ teaching performance. Perspectives on Medical Education 2015; 4(5): 264-267. https://doi.org/10.1007/S40037-015-0215-7 DOI: https://doi.org/10.1007/S40037-015-0215-7

Constantinou C, Wijnen-Meijer M. Student evaluations of teaching and the development of a comprehensive measure of teaching effectiveness for medical Schools. BMC Medical Education 2022; 22(113): 1-12. https://doi.org/10.1186/s12909-022-03148-6 DOI: https://doi.org/10.1186/s12909-022-03148-6

Dodeen H. Validity, Reliability, and Potential Bias of Short Forms of Students’ Evaluation of Teaching: The Case of UAE University. Educational Assessment 2013; 18(4): 235-250. https://doi.org/10.1080/10627197.2013.846670 DOI: https://doi.org/10.1080/10627197.2013.846670

Abdallah A, Balla B. Students' Evaluation of Teaching Effectiveness: Level of Acceptance, Implementation, and Causes for Concern (A Case Study of Saudi Faculty Members at Jeddah University-Kholais Branch). International Journal of English Language Teaching 2022; 10(3): 24-36. https://doi.org/10.37745/ijelt.13/vol10no2pp.24-36 DOI: https://doi.org/10.37745/ijelt.13/vol10no2pp.24-36

Pan G, Shankararaman V, Koh K, Gan S. Students’ evaluation of teaching in the project-based learning programme: An instrument and a development process. The International Journal of Management Education 2021; 19(2): 100501.‏ https://doi.org/10.1016/j.ijme.2021.100501 DOI: https://doi.org/10.1016/j.ijme.2021.100501

Kogan LR, Schoenfeld-Tacher R, Hellyer PW. Student evaluations of teaching: perceptions of faculty based on gender, position, and rank. Teaching in Higher Education 2010; 15(6): 623-636. https://doi.org/10.1080/13562517.2010.491911 DOI: https://doi.org/10.1080/13562517.2010.491911

Park E, Dooris J. Predicting student evaluations of teaching using decision tree analysis. Assessment & Evaluation in Higher Education 2020; 45(5): 776-793. https://doi.org/10.1080/02602938.2019.1697798 DOI: https://doi.org/10.1080/02602938.2019.1697798

Urrutia-Aguilar M, Sánchez-Mendiola M, Guevara-Guzmán R, Martínez-González A. Comprehensive Assessment of Teaching Performance in Medical Education. Procedia - Social and Behavioral Sciences 2014; (141): 252-259. https://doi.org/10.1016/j.sbspro.2014.05.044

Schönrock-Adema J, Boendermaker P, Remmelts P. Opportunities for the CTEI: disentangling frequency and quality in evaluating teaching behaviours. Perspectives on Medical Education 2012; (1): 172-179. https://doi.org/10.1007/S40037-012-0023-2 DOI: https://doi.org/10.1007/S40037-012-0023-2

Wright S, Jenkins-Guarnieri M. Student evaluations of teaching: Combining the meta-analyses and demonstrating further evidence for effective use. Assessment & Evaluation in Higher Education 2012; 37(6): 683-699.‏ https://doi.org/10.1080/02602938.2011.563279 DOI: https://doi.org/10.1080/02602938.2011.563279

Uttl B, White C, Gonzalez D. Meta-analysis of faculty’s teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation 2016; (54): 22-42. https://doi.org/10.1016/j.stueduc.2016.08.007 DOI: https://doi.org/10.1016/j.stueduc.2016.08.007

Oudkerk Pool A, Jaarsma A, Driessen E, Govaerts M. Student perspectives on competency-based portfolios: Does a portfolio reflect their competence development? Perspectives on Medical Education 2020; (9)1: 66-172. https://doi.org/10.1007/S40037-020-00571-7 DOI: https://doi.org/10.1007/S40037-020-00571-7

Müller T, Montano D, Poinstingl H, et al. Evaluation of large-group lectures in medicine – development of the SETMED-L (Student Evaluation of Teaching in MEDical Lectures) questionnaire. Müller et al. BMC Medical Education 2017; (17): 137. https://doi.org/10.1186/s12909-017-0970-8 DOI: https://doi.org/10.1186/s12909-017-0970-8

Ahmed M. Are good attributes of medical teachers more important than the learning style: a glimpse into the future of medical education and learning. Journal of Public Health and Emergency 2018; (2). https://doi.org/10.21037/jphe.2018.05.01 DOI: https://doi.org/10.21037/jphe.2018.05.01

Engbers R, de Caluwé L, Stuyt P, Fluit C, Bolhuis S. Towards organizational development for sustainable high-quality medical teaching. Perspectives on Medical Education 2013; (2): 28-40. https://doi.org/10.1007/S40037-013-0043-6 DOI: https://doi.org/10.1007/S40037-013-0043-6

Worthington A. The Impact of Student Perceptions and Characteristics on Teaching Evaluations: A case study in finance education. Assessment & Evaluation in Higher Education 2002; 27(1): 49-64. https://doi.org/10.1080/02602930120105054 DOI: https://doi.org/10.1080/02602930120105054

Almakadma A, Fawzy N, Baqal O, Kamada S. Perceptions and attitudes of medical students towards student evaluation of teaching: A cross-sectional study. Medical Education Online 2023; (28)1: 2220175. https://doi.org/10.1080/10872981.2023.2220175 DOI: https://doi.org/10.1080/10872981.2023.2220175

Haris S, Jamil B, Haris M, Deeba F, Khan MJ, and Khan IZ. Factors Affecting Students Perception towards Faculty Evaluation of Teaching at Nowshera Medical College. The Professional Medical Journal 2022; 29(2): 258-264. https://doi.org/10.29309/TPMJ/2022.29.02.6407 DOI: https://doi.org/10.29309/TPMJ/2022.29.02.6407

Lawrence J. Student Evaluations of Teaching are Not Valid American Association of University Professors. Epub 2018.

Singh S, Pai D, Sinha N, Kaur A, Soe H, Barua A. Qualities of an effective teacher: what do medical teachers think? BMC Medical Education 2013; 13(1). https://doi.org/10.1186/1472-6920-13-128 DOI: https://doi.org/10.1186/1472-6920-13-128

Urrutia-Aguilar M, Sánchez-Mendiola M, Guevara-Guzmán R, Martínez-González A. Comprehensive Assessment of Teaching Performance in Medical Education. Procedia - Social and Behavioral Sciences 2014; (141): 252-259. https://doi.org/10.1016/j.sbspro.2014.05.044 DOI: https://doi.org/10.1016/j.sbspro.2014.05.044

Dimitrov D. Testing for factorial invariance in the context of construct validation. Measurement and Evaluation in Counseling and Development 2010; 43(2): 121-149. https://doi.org/10.1177/0748175610373459 DOI: https://doi.org/10.1177/0748175610373459

Sánchez T, Gilar-Corbi R, Castejón J, Vidal J, León J. Students’ evaluation of teaching and their academic achievement in a higher education institution of Ecuador. Frontiers in Psychology 2020; 11(233).‏ https://doi.org/10.3389/fpsyg.2020.00233 DOI: https://doi.org/10.3389/fpsyg.2020.00233

Boring A. Gender biases in student evaluations of teachers. Journal of Public Economics 2017; 145(13): 27-41. https://doi.org/10.1016/j.jpubeco.2016.11.006 DOI: https://doi.org/10.1016/j.jpubeco.2016.11.006

Chen G, Watkins D. Stability and correlates of student evaluations of teaching at a Chinese university. Assessment & Evaluation in Higher Education 2010; 35(6): 675-685. https://doi.org/10.1080/02602930902977715 DOI: https://doi.org/10.1080/02602930902977715

Wolbring T, Riordan P. How beauty works. Theoretical mechanisms and two empirical applications on students' evaluation of teaching. Social Science Research 2016; 5(7): 253-272. https://doi.org/10.1016/j.ssresearch.2015.12.009 DOI: https://doi.org/10.1016/j.ssresearch.2015.12.009

Harnish R, Bridges K. Effect of syllabus tone: Students’ perceptions of instructor and course. Social Psychology of Education 2011; 14(3): 319-330.‏ https://doi.org/10.1007/s11218-011-9152-4 DOI: https://doi.org/10.1007/s11218-011-9152-4

Park B, Cho J. How does grade inflation affect student evaluation of teaching? Assessment & Evaluation in Higher Education 2022. https://doi.org/10.1080/02602938.2022.2126429 DOI: https://doi.org/10.1080/02602938.2022.2126429

Krammer G, Pflanzl B, Lenske G, Mayr J. Assessing quality of teaching from different perspectives: Measurement invariance across teachers and classes. Educational Assessment 2021; 26(2): 88-103. https://doi.org/10.1080/10627197.2020.1858785 DOI: https://doi.org/10.1080/10627197.2020.1858785

Van de Schoot R, Lugtig P, Hox J. A checklist for testing measurement invariance. European Journal of Developmental Psychology 2012; 9(4): 486-492.‏ https://doi.org/10.1080/17405629.2012.686740 DOI: https://doi.org/10.1080/17405629.2012.686740

Asparouhov T, Muthén B. Multiple-group factor analysis alignment. Structural Equation Modeling: A Multidisciplinary Journal 2014; 21(4): 495-508. https://doi.org/10.1080/10705511.2014.919210 DOI: https://doi.org/10.1080/10705511.2014.919210

Putnick D, Bornstein M. Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review 2016; (41): 71-90.‏ https://doi.org/10.1016/j.dr.2016.06.004

Rutkowski L, Svetina D. Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement 2014; 74(1): 31-57. https://doi.org/10.1177/0013164413498257

Chen F. Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal 2007; 14(3): 464-504.‏ https://doi.org/10.1080/10705510701301834 DOI: https://doi.org/10.1080/10705510701301834

Bazán-Ramírez A, Pérez-Morán J, Bernal-Baldenebro B. Criteria for teaching performance in psychology: invariance according to age, sex, and academic stage of Peruvian students. Frontiers in Psychology 2021; (12): 4816.‏ https://doi.org/10.3389/fpsyg.2021.764081 DOI: https://doi.org/10.3389/fpsyg.2021.764081

Kalender I, Berberoğlu G. The measurement invariance of University students’ ratings of instruction. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi 2019; 34(2): 402-417.‏

Putnick D, Bornstein M. Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review 2016; (41): 71-90.‏ https://doi.org/10.1016/j.dr.2016.06.004 DOI: https://doi.org/10.1016/j.dr.2016.06.004

Pokropek A, Davidov E, Schmidt P. A Monte Carlo simulation study to assess the appropriateness of traditional and newer approaches to test for measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal 2019; 26(5): 724-744. https://doi.org/10.1080/10705511.2018.1561293 DOI: https://doi.org/10.1080/10705511.2018.1561293

Walsh K. Medical education research: is participation fair? Perspectives on Medical Education 2014; (3): 379-382. https://doi.org/10.1007/S40037-014-0120-5 DOI: https://doi.org/10.1007/S40037-014-0120-5

Rutkowski L, Svetina D. Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement 2014; 74(1): 31-57. https://doi.org/10.1177/0013164413498257 DOI: https://doi.org/10.1177/0013164413498257

Steinmetz H. Analyzing observed composite differences across groups. Methodology 2013; 9(1): 1-12. https://doi.org/10.1027/1614-2241/a000049 DOI: https://doi.org/10.1027/1614-2241/a000049