Developing and Validating the Claremont Evaluative Thinking Scale
Article Sidebar
Main Article Content
Abstract
Evaluation scholars and practitioners have argued that having more people who can think evaluatively is “essential” for successful evaluation practice, organizational improvement, and even healthy democracies. However, empirical evidence backing those claims is scant. Making a case for investing more resources into building people’s capacity to think evaluatively requires a better understanding of its impact and added value. We need a way to measure it. While two scales to measure evaluative thinking exist, both have shortcomings and neither has been used to build a body of evidence concerning the contributions of evaluative thinking to the evaluation field and beyond. A two-stage, mixed-methods study was conducted to create a new, reliable and valid evaluative thinking scale. Stage One identified dimensions of evaluative thinking through literature review and expert interviews. Stage Two generated and refined items to measure these dimensions, confirmed face validity through focus groups, and administered the refined items to 250 random, yet screened, participants. Factor analysis revealed six factors with strong item loadings, and a Cronbach’s alpha test confirmed reliability (α = 0.96). This study enhances our capacity to measure evaluative thinking and lays the groundwork for future research into understanding its impact more deeply.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Archibald, T., Sharrock, G., Buckley, J., & Cook, N. (2016). Assumptions, conjectures, and other miracles: The application of evaluative thinking to theory of change models in community development. Evaluation and Program Planning, 59, 119–127. https://doi.org/10.1016/j.evalprogplan.2016.05.015 DOI: https://doi.org/10.1016/j.evalprogplan.2016.05.015
Buckley, J. & Archibald, T. (2011). The Evaluative Thinking Inventory. Cornell University.
Buckley, J., Archibald, T., Hargraves, M., & Trochim, W. M. (2015). Defining and teaching evaluative thinking: Insights from research on critical thinking. American Journal of Evaluation, 36(3), 375–388. DOI: https://doi.org/10.1177/1098214015581706
Chauveron, L. M., Urban, J. B., Samtani, S., Cox, M., Moorman, L., Hargraves, M., Buckley, J., & Linver, M. (2021). Promoting evaluation in youth character development through enhanced evaluation capacity building: Empirical findings from the PACE Project. New Directions for Evaluation, 169, 79–95. https://doi.org/10.1002/ev.20447 DOI: https://doi.org/10.1002/ev.20447
Crano, W. D., Brewer, M. B., & Lac, A. (2015). Principles and methods of social research (3rd ed). Routledge. http://site.ebrary.com/id/10921638 DOI: https://doi.org/10.4324/9781315768311
Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (Third edition). SAGE.
Flora, D. B., & Flake, J. K. (2017). The purpose and practice of exploratory and confirmatory factor analysis in psychological research: Decisions for scale development and validation. Canadian Journal of Behavioural Science, 49(2), 78–88. http://dx.doi.org/10.1037/cbs0000069 DOI: https://doi.org/10.1037/cbs0000069
Gates, E. F. (2018). Toward valuing with critical systems heuristics. American Journal of Evaluation, 39(2), 201–220. DOI: https://doi.org/10.1177/1098214017703703
Gorsuch, R. L. (1983). Factor analysis (2nd ed.). L. Erlbaum Associates.
Hargraves, M., Buckley, J., Urban, J. B., Linver, M. R., Chauveron, L. M., Samtani, S.,
Moorman, L. (2021). Resonance, stickiness, and the value propositions of Evaluation Capacity Building: Key takeaways and future directions. New Directions for Evaluation, 2021(169), 97-116 DOI: https://doi.org/10.1002/ev.20442
Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7(2), 191–205. https://doi.org/10.1177/1094428104263675 DOI: https://doi.org/10.1177/1094428104263675
Howard, M. C. (2016). A review of exploratory factor analysis decisions and overview of current practices: What are we doing and how can we improve? International Journal of Human-Computer Interaction, 32(1), 51–62. http://dx.doi.org/10.1080/10447318.2015.1087664 DOI: https://doi.org/10.1080/10447318.2015.1087664
Lai, E. R. (2011). Critical thinking: A literature review. Pearson's Research Reports, 6(1), 40–41.
Leviton, L. C. (2014). Some underexamined aspects of evaluation capacity building. American Journal of Evaluation, 35(1), 90–94. https://doi.org/10.1177/1098214013502844 DOI: https://doi.org/10.1177/1098214013502844
Maxwell, J. A. (2013), Qualitative research design: An interactive approach (3rd ed.). SAGE Publications.
McIntosh, J. S., Buckley, J., & Archibald, T. (2020). Refining and measuring the construct of evaluative thinking: An exploratory factor analysis of the Evaluative Thinking Inventory. Journal of MultiDisciplinary Evaluation, 16, 104–117. DOI: https://doi.org/10.56645/jmde.v16i34.591
Papadas, K.-K., Avlonitis, G. J., & Carrigan, M. (2017). Green marketing orientation: Conceptualization, scale development and validation. Journal of Business Research, 80, 236–246. https://doi.org/10.1016/j.jbusres.2017.05.024 DOI: https://doi.org/10.1016/j.jbusres.2017.05.024
Patton, M. P. (2022, February 28). Research Focused Approaches to Evaluation [Lecture]. Comparative evaluation theories, Claremont Graduate University.
Patton, M. Q. (2018). A historical perspective on the evolution of evaluative thinking. New Directions for Evaluation, 158, 11–28. https://doi.org/10.1002/ev.20325 DOI: https://doi.org/10.1002/ev.20325
Schwandt, T. A. (2018). Evaluative thinking as a collaborative social practice: The case of boundary judgment making: Evaluative thinking as a collaborative social practice. New Directions for Evaluation, 158, 125–137. https://doi.org/10.1002/ev.20318 DOI: https://doi.org/10.1002/ev.20318
Scriven, M. (1991). Evaluation thesaurus (4th ed.). Sage Publications.
Stockdill, S. H., Baizerman, M., & Compton, D. W. (2002). Toward a Definition of the ECB Process: A Conversation with the ECB Literature. New Directions for Evaluation, 93, 1–25. DOI: https://doi.org/10.1002/ev.38
Symonette, H. (2004). Walking pathways toward becoming a culturally competent evaluator: Boundaries, borderlands, and border crossings. New Directions for Evaluation, 102, 95–109. https://doi.org/10.1002/ev.118 DOI: https://doi.org/10.1002/ev.118
Tan, L., Kocsis, A., & Burry, J. (2023). Advancing Donald Schön’s Reflective Practitioner: Where to Next? Design Issues, 39(3), 3–18. https://doi.org/10.1162/desi_a_00722 DOI: https://doi.org/10.1162/desi_a_00722
The Bruner Foundation. (2014–2015). Effectiveness initiatives: Evaluative thinking. http://www.evaluativethinking.org/evalthink.html
Tovey, T. L. S. & Archibald, T. (2023). The relationship between reflective practice, evaluative thinking, and practical wisdom. In M. Hurteau & T. Archibald (Eds.), Practical Wisdom for an Ethical Evaluation Practice (pp. 87-101). Information Age Publishing. DOI: https://doi.org/10.1108/979-8-88730-088-7-20251011
Vo, A. T. (2013). Toward a definition of Evaluative Thinking [UCLA]. https://escholarship.org/uc/item/26t5x5f6
Vo, A.T. and Archibald, T. (2018), Editors' Notes. Evaluation, 2018: 7-9. https://doi-org.ccl.idm.oclc.org/10.1002/ev.20321 DOI: https://doi.org/10.1002/ev.20321
Vo, A. T., Schreiber, J. S., & Martin, A. (2018). Toward a conceptual understanding of Evaluative Thinking. New Directions for Evaluation, 158, 29–47. https://doi.org/10.1002/ev.20324 DOI: https://doi.org/10.1002/ev.20324
Wingate, L. A., Robertson, K., FitzGerald, M., Rucks, L., Tsuzaki, T., Clasen, C., & Schwob, J. (2022). Thinking outside the self-report: Using evaluation plans to assess evaluation capacity building. American Journal of Evaluation, 43(4), 515–538. https://doi.org/10.1177/10982140211062884 DOI: https://doi.org/10.1177/10982140211062884