Determining expert-derived weights for Kirkpatrick’s evaluation model levels and effectiveness grading cut off points in assessing skill based agricultural extension training courses

Document Type : Research Paper

Author

Socio-Economic and Agricultural Extension Research Department, Fars Agricultural and Natural Resources Research and Education Center, AREEO, Shiraz, I. R. Iran

Abstract

Kirkpatrick’s four‑level model (reaction, learning, behavior, and results) is the most widely used framework for training effectiveness evaluation. In recognizing a lack of an evidence-based weighting framework for Kirkpatrick’s four evaluation levels, this study developed a practical weighting model based on expert consensus to improve the evaluation of agricultural extension training programs. This study aimed to determine expert-derived weights for the four levels of Kirkpatrick’s evaluation model and to establish cut‑off points for the effectiveness grade of the composite effectiveness score for agricultural extension training courses. As a quantitative survey study, the target population consisted of Ph.D. specialists in agricultural extension and education working in universities, the Agricultural Research, Education and Extension Organization (AREEO), and Agricultural Extension Coordination Offices. A researcher‑developed questionnaire was sent to 64 specialists. Fifty-one complete responses were received (response rate 79.7%). Data were analyzed using descriptive statistics (frequency, percentage) and inferential statistics (Pearson’s chi‑square test). The findings showed no significant difference among university professors and AREEO personnel. The two groups had homogeneous opinions. Based on the weighted average method, the adjusted recommended weights for reaction, learning, behavior, and results were 15%, 20%, 35%, and 30%, respectively. The proposed effectiveness grading cut‑offs were: score > 80 (high), 60–80 (moderate), 40–60 (weak), and < 40 (non-effective), which fully matched the specialists’ modal choice. The results of this study can provide a layout for standardizing effectiveness evaluation in agricultural extension training courses in Iran.

Graphical Abstract

Determining expert-derived weights for Kirkpatrick’s evaluation model levels and effectiveness grading cut off points in assessing skill based agricultural extension training courses

Keywords


Bakhshi Jahromi, A., & Heydari, M. (2022). Effectiveness of in‑service training courses for employees of Fars Agricultural Jihad Organization. Proceedings of the 9th National Congress of Agricultural Extension, Natural Resources and Sustainable Environment, September 6‑8, Tabriz, pp. 1‑12. (In Persian)
Allen, M. J., & Yen, W. M. (2001). Introduction to measurement theory. Long Grove, Illinois: Waveland Press.
Blume, B. D., Ford, J. K., Baldwin, T. T., & Huang, J. L. (2010). Transfer of training: A meta‐analytic review. Journal of Management, 36(4), 1065-1105. https://doi.org/10.1177/0149206309352880
Cohen, R. J., & Swerdlik, M. E. (2021). Psychological testing and assessment (9th ed.). New York: McGraw‑Hill.
Faisal-E-Alam, M., & Islam, A. R. M. T. (2025). Exploring the hierarchical framework of the Kirkpatrick model in training evaluation from a developing country. Evaluation and Program Planning, 113, 102696. https://doi.org/10.1016/j.evalprogplan.2025.102696
Hsu, C. -C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation, 12(10), 1-8. https://doi.org/10.7275/pdz9-th90
Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Kirkpatrick’s four levels of training evaluation. Washington, DC: ATD Press.
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels (3rd ed.). Berrett-Koehler Publishers.
Nitko, A. J., & Brookhart, S. M. (2014). Educational assessment of students (7th ed.). U. K: Pearson Education.
Norman, G. (2010). Likert scales, levels of measurement and the “laws” of statistics. Advances in Health Sciences Education, 15(5), 625-632. https://doi.org/10.1007/s10459-010-9222-y
Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: An example, design considerations and applications. Information & Management, 42(1), 15-29. https://doi.org/10.1016/j.im.2003.11.002
Paul, S., Burman, R. R., & Singh, R. (2024). Training effectiveness evaluation: Advancing a Kirkpatrick model based composite framework. Evaluation and Program Planning, 107, 102494. https://doi.org/10.1016/j.evalprogplan.2024.102494
Praslova, L. (2010). Adaptation of Kirkpatrick’s four level model of training criteria to assessment of learning outcomes and program evaluation in higher education. Educational Assessment, Evaluation and Accountability, 22(3), 215-225. https://doi.org/10.1007/s11092-010-9098-7
Sathishwaran, R., Helen, S., Bonny, B. P., & Darsana, S. (2025). Structural equation modelling analysis of the Kirkpatrick training evaluation model. The Journal of Agricultural Education and Extension, 1-17. https://doi.org/10.1080/1389224X.2025.2573498
Taylor, M., & Bhasme, S. (2018). Model farmers, extension networks and the politics of agricultural knowledge transfer. Journal of Rural Studies, 64, 1-10. 10.1016/j.jrurstud.2018.09.015
te Lintelo, D. J. H., Haddad, L. J., Leavy, J., & Lakshman, R. (2014). Measuring the commitment to reduce hunger: A hunger reduction commitment index. Food Policy, 44, 115-128. https://doi.org/10.1016/j.foodpol.2013.11.005
Tiyawongsuwan, S., Lawthong, N., & Kanjanawasee, S. (2013). Standard setting for educational evaluation: concepts, methods and applications. Kasetsart Journal of Social Sciences, 34(2), 225-240.
Zardari, N. H., Ahmed, K., Shirazi, S. M., & Yusop, Z. B. (2015). Weighting methods and their effects on multi-criteria decision Making model outcomes in water resources management. Springer.