Comparing Dichotomous and Polytomous Instruments in Cdm-Based Swimming Technique Assessment
Volume 16, Issue 2 (2026)
Volume 16, Issue 2 (2026)
Comparing Dichotomous and Polytomous Instruments in Cdm-Based Swimming Technique Assessment
Abstract:
Assessing complex motor skills across physical performance domains poses distinct methodological challenges for applying the Cognitive Diagnostic Model (CDM), particularly regarding the response format of observational instruments. This study compared inter-rater reliability, model fit, diagnostic specificity, and criterion validity between dichotomous and polytomous instruments for assessing front crawl technique. A total of 280 physical education students from Universitas Negeri Surabaya participated. The dichotomous instrument (10 items, three attributes: Arm Action, Body & Leg, Breath & Synchronization) was adapted from the Crawl Technique Observation Sheet and analyzed using GDINA. The polytomous instrument (20 items, four dimensions) was analyzed using pGDINA with a three-attribute Q-matrix, after Body Position and Leg Action were mapped into a single Body & Leg attribute based on biomechanical justification, structural equivalence, and sample adequacy. The dichotomous instrument showed higher inter-rater reliability (Fleiss’ κ = 0.663) than the polytomous instrument (ICC = 0.538), with a Classification Accuracy of 0.924 and higher validity against swimming speed (ρ = 0.507 vs. 0.288). The polytomous instrument produced higher diagnostic entropy (2.362 vs. 1.568 bits) and a more even latent profile distribution (largest profile 40.7% vs. 62.5%), indicating better diagnostic specificity. Cross-instrument profile consistency was low (Cohen’s κ = 0.370), with 31.4% of borderline students classified differently by both instruments. A trade-off emerged: the dichotomous instrument excelled in reliability, classification accuracy, and criterion validity, whereas the polytomous instrument excelled in diagnostic depth, confirming that CDM applications in motor skill assessment face challenges distinct from cognitive education contexts.
Keywords:
Cognitive Diagnostic Model; GDINA; polytomous instrument; inter-rater reliability; front crawl swimming
Full Text:
References:
Aiken, L. R. (1985). Three Coefficients for Analyzing the Reliability and Validity of Ratings. Educational and Psychological Measurement, 45(1), 131–142. https://doi.org/10.1177/0013164485451012 Chen, J., & de la Torre, J. (2018). Introducing the general polytomous diagnosis modeling framework. Frontiers in Psychology, 9. https://doi. org/10.3389/FPSYG.2018.01474 Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi. org/10.1177/001316446002000104 Collares, C. F. (2022). Cognitive diagnostic modelling in healthcare professions education: an eye-opener. Advances in Health Sciences Education, 27(2), 427–440. https://doi.org/10.1007/S10459-022-10093-Y de la Torre, J. (2011). The Generalized DINA Model Framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/S11336-011-9207-7 de la Torre, J., & Akbay, L. (2019). Implementation of cognitive diagnosis modeling using the GDINA r package. Eurasian Journal of Educational Research, 171–192. https://eric.ed.gov/?id=EJ1211628 de Medeiros Vidal, J., Tucher, G., Nogueira, L., Novaes, R. C., de Souza Vale, R. G., de Castro, M. O. R., da Silva Medeiros, N. G., & de Cassio Costa Telles, S. (2021). Crawl technique observation sheet for beginning swimmers: an evaluation proposal for swimming teachers. Motriz: Revista de Educação Física, 27, e1021016920. https://doi.org/10.1590/S1980-65742021016920 Gamer, M., Lemon, J., & Singh, I. F. P. (2019). irr: Various coefficients of interrater reliability and agreement. CRAN. https://cran.r-project. org/package=irr George, A. C., Robitzsch, A., Kiefer, T., Groß, J., & Ünlü, A. (2016). The R package CDM for cognitive diagnosis models. Journal of Statistical Software, 74(2), 1–24. https://doi.org/10.18637/jss.v074.i02 Gudicha, D. W., Tekle, F. B., & Vermunt, J. K. (2016). Power and Sample Size Computation for Wald Tests in Latent Class Models. Journal of Classification, 33(1), 30–51. https://doi.org/10.1007/S00357-016-9199-1 Jürimäe, J., Haljaste, K., Cicchella, A., Lätt, E., Purge, P., Leppik, A., & Jürimäe, T. (2007). Analysis of Swimming Performance from Physical, Physiological, and Biomechanical Parameters in Young Swimmers. Pediatric Exercise Science, 19(1), 70–81. https://doi.org/10.1123/ PES.19.1.70 Landis, J. R., & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159. https://doi. org/10.2307/2529310 Lei, P. W., & Li, H. (2016). Performance of Fit Indices in Choosing Correct Cognitive Diagnostic Models and Q-Matrices. Applied Psychological Measurement, 40(6), 405–417. https://doi.org/10.1177/0146621616647954 Ma, W., & de la Torre, J. (2016). A sequential cognitive diagnosis model for polytomous responses. The British Journal of Mathematical and Statistical Psychology, 69(3), 253–275. https://doi.org/10.1111/BMSP.12070 Ma, W., & de la Torre, J. (2020). GDINA: An R package for cognitive diagnosis modeling. Journal of Statistical Software, 93(14), 1–26. https:// doi.org/10.18637/jss.v093.i14 Maglischo, E. W. (2003). Swimming fastest. Human Kinetics. Maydeu-Olivares, A., & Joe, H. (2006). Limited Information Goodness-of-fit Testing in Multidimensional Contingency Tables. Psychometrika, 71(4), 713–732. https://doi.org/10.1007/S11336-005-1295-9 Meng, Y., Wang, Y., & Zhao, N. (2023). Cognitive diagnostic assessment of EFL learners’ listening barriers through incorrect responses. Frontiers in Psychology, 14, 1126106. https://doi.org/10.3389/FPSYG.2023.1126106 Moskal, B. M., & Leydens, J. A. (2000). Scoring Rubric Development: Validity and Reliability. Practical Assessment, Research, and Evaluation, 7(1). https://doi.org/10.7275/Q7RM-GG74 Popham, W. J. (2003). Test better, teach better: The instructional role of assessment. ASCD. Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press. Sampaio Peres, M. V., Nogueira, F., Ramos, W., Cardoso, T., Nogueira, L., & Tucher, G. (2023). Effect of standardization of observation and evaluation criteria of the front crawl arm stroke technique on inter- and intra-rater reliability. Journal of Physical Education, 34(1). https://doi.org/10.4025/jphyseduc.v34i1.3416 Sorrel, M. A., Escudero, S., Nájera, P., Kreitchmann, R. S., & Vázquez-Lira, R. (2023). Exploring Approaches for Estimating Parameters in Cognitive Diagnosis Models with Small Sample Sizes. Psych, 5(2), 336–349. https://doi.org/10.3390/PSYCH5020023 Stemler, S. E. (2004). A Comparison of Consensus, Consistency, and Measurement Approaches to Estimating Interrater Reliability. Practical Assessment, Research, and Evaluation, 9(1). https://doi.org/10.7275/96JP-XZ07 Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677–680. https://doi.org/10.1126/SCIENCE.103.2684.677 Zumbo, B. D., & Kroc, E. (2019). A Measurement Is a Choice and Stevens’ Scales of Measurement Do Not Help Make It: A Response to Chalmers. Educational and Psychological Measurement, 79(6), 1184–1197. https://doi.org/10.1177/0013164419844305
