Zum Hauptinhalt springen

Why we need to abandon fixed cutoffs for goodness-of-fit indices: An extensive simulation and possible solutions.

Groskurth, K ; Bluemke, M ; et al.
In: Behavior research methods, Jg. 56 (2024-04-01), Heft 4, S. 3891-3914
Online academicJournal

Titel:
Why we need to abandon fixed cutoffs for goodness-of-fit indices: An extensive simulation and possible solutions.
Autor/in / Beteiligte Person: Groskurth, K ; Bluemke, M ; Lechner, CM
Link:
Zeitschrift: Behavior research methods, Jg. 56 (2024-04-01), Heft 4, S. 3891-3914
Veröffentlichung: 2010- : New York : Springer ; <i>Original Publication</i>: Austin, Tex. : Psychonomic Society, c2005-, 2024
Medientyp: academicJournal
ISSN: 1554-3528 (electronic)
DOI: 10.3758/s13428-023-02193-3
Schlagwort:
  • Social Sciences methods
  • Social Sciences standards
  • Behavioral Sciences methods
  • Behavioral Sciences standards
  • Chi-Square Distribution
  • Computer Simulation standards
  • Factor Analysis, Statistical
Sonstiges:
  • Nachgewiesen in: MEDLINE
  • Sprachen: English
  • Publication Type: Journal Article
  • Language: English
  • [Behav Res Methods] 2024 Apr; Vol. 56 (4), pp. 3891-3914. <i>Date of Electronic Publication: </i>2023 Aug 28.
  • MeSH Terms: Computer Simulation* / standards ; Factor Analysis, Statistical* ; Social Sciences / methods ; Social Sciences / standards ; Behavioral Sciences / methods ; Behavioral Sciences / standards ; Chi-Square Distribution
  • References: Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling, 13(2), 186–203. https://doi.org/10.1207/s15328007sem1302_2. (PMID: 10.1207/s15328007sem1302_2) ; Beauducel, A., & Wittmann, W. W. (2005). Simulation study on fit indexes in CFA based on data with slightly distorted simple structure. Structural Equation Modeling, 12(1), 41–75. https://doi.org/10.1207/s15328007sem1201_3. (PMID: 10.1207/s15328007sem1201_3) ; Beierl, E. T., Bühner, M., & Heene, M. (2018). Is that measure really one-dimensional? Methodology, 14(4), 188–196. https://doi.org/10.1027/1614-2241/a000158. (PMID: 10.1027/1614-2241/a000158) ; Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246. https://doi.org/10.1037/0033-2909.107.2.238. (PMID: 10.1037/0033-2909.107.2.2382320703) ; Bentler, P. M. (1995). EQS structural equations program manual. Multivariate Software. ; Bilsky, W., Janik, M., & Schwartz, S. H. (2011). The structural organization of human values-evidence from three rounds of the European Social Survey (ESS). Journal of Cross-Cultural Psychology, 42(5), 759–776. https://doi.org/10.1177/0022022110362757. (PMID: 10.1177/0022022110362757) ; Blanca, M. J., Arnau, J., López-Montiel, D., Bono, R., & Bendayan, R. (2013). Skewness and kurtosis in real data samples. Methodology, 9(2), 78–84. https://doi.org/10.1027/1614-2241/a000057. (PMID: 10.1027/1614-2241/a000057) ; Bollen, K. A. (1989). Structural equations with latent variables. Wiley. (PMID: 10.1002/9781118619179) ; Bonett, D. G. (2012). Replication-extension studies. Current Directions in Psychological Science, 21(6), 409–412. https://doi.org/10.1177/0963721412459512. (PMID: 10.1177/0963721412459512) ; Boomsma, A. (2013). Reporting Monte Carlo studies in structural equation modeling. Structural Equation Modeling, 20(3), 518–540. https://doi.org/10.1080/10705511.2013.797839. (PMID: 10.1080/10705511.2013.797839) ; Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504. https://doi.org/10.1080/10705510701301834. (PMID: 10.1080/10705510701301834) ; Chen, F., Curran, P. J., Bollen, K. A., Kirby, J., & Paxton, P. (2008). An empirical evaluation of the use of fixed cutoff points in RMSEA test statistic in structural equation models. Sociological Methods & Research, 36(4), 462–494. https://doi.org/10.1177/0049124108314720. (PMID: 10.1177/0049124108314720) ; Clark, L. A., & Watson, D. (2019). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 31(12), 1412–1427. https://doi.org/10.1037/pas0000626. (PMID: 10.1037/pas0000626308962126754793) ; Comrey, A. L., & Lee, H. B. (1992). Interpretation and Application of Factor Analytic Results (2nd ed.). Lawrence Erlbaum Associates. ; DiStefano, C., McDaniel, H. L., Zhang, L., Shi, D., & Jiang, Z. (2019). Fitting large factor analysis models with ordinal data. Educational and Psychological Measurement, 79(3), 417–436. https://doi.org/10.1177/0013164418818242. (PMID: 10.1177/001316441881824231105317) ; Fan, X., & Sivo, S. A. (2007). Sensitivity of fit indices to model misspecification and model types. Multivariate Behavioral Research, 42(3), 509–529. https://doi.org/10.1080/00273170701382864. (PMID: 10.1080/00273170701382864) ; Garrido, L. E., Abad, F. J., & Ponsoda, V. (2016). Are fit indices really fit to estimate the number of factors with categorical variables? Some cautionary findings via Monte Carlo simulation. Psychological Methods, 21(1), 93–111. https://doi.org/10.1037/met0000064. (PMID: 10.1037/met000006426651983) ; Groskurth, K., Nießen, D., Rammstedt, B., & Lechner, C. M. (2021). An English-language adaptation and validation of the political efficacy short scale (PESS). Measurement Instruments for the Social Sciences, 3, 1. https://doi.org/10.1186/s42409-020-00018-z. (PMID: 10.1186/s42409-020-00018-z) ; Hancock, G. R., & Mueller, R. O. (2011). The reliability paradox in assessing structural relations within covariance structure models. Educational and Psychological Measurement, 71(2), 306–324. https://doi.org/10.1177/0013164410384856. (PMID: 10.1177/0013164410384856) ; Hayduk, L. (2014). Seeing perfectly fitting factor models that are causally misspecified: Understanding that close-fitting models can be worse. Educational and Psychological Measurement, 74(6), 905–926. https://doi.org/10.1177/0013164414527449. (PMID: 10.1177/0013164414527449) ; Heene, M., Hilbert, S., Draxler, C., Ziegler, M., & Bühner, M. (2011). Masking misfit in confirmatory factor analysis by increasing unique variances: A cautionary note on the usefulness of cutoff values of fit indices. Psychological Methods, 16(3), 319–336. https://doi.org/10.1037/a0024917. (PMID: 10.1037/a002491721843002) ; Hu, L.-T., & Bentler, P. M. (1999). Cutoff criteria for fit indices in covariance structure model: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118. (PMID: 10.1080/10705519909540118) ; Jackson, D. L., Gillaspy, J. A., Jr., & Purc-Stephenson, R. (2009). Reporting practices in confirmatory factor analysis: an overview and some recommendations. Psychological Methods, 14(1), 6–23. https://doi.org/10.1037/a0014694. (PMID: 10.1037/a001469419271845) ; Jöreskog, K. G., & Sörbom, D. (1993). LISREL 8: Structural equation modeling with the SIMPLIS command language. Scientific Software International. ; Kenny, D. A., & McCoach, D. B. (2003). Effect of the number of variables on measures of fit in structural equation modeling. Structural Equation Modeling, 10(3), 333–351. https://doi.org/10.1207/S15328007SEM1003_1. (PMID: 10.1207/S15328007SEM1003_1) ; Kenny, D. A., Kaniskan, B., & McCoach, D. B. (2015). The performance of RMSEA in models with small degrees of freedom. Sociological Methods & Research, 44(3), 486–507. https://doi.org/10.1177/0049124114543236. (PMID: 10.1177/0049124114543236) ; Kim, J. H., Lee, J., Richardson, T. V., Lee, D. H., McMahon, B. T., Kim, H., & Sametz, R. R. (2022). Psychometric validation of adapted inventory of virtues and strengths. Rehabilitation Counseling Bulletin, 65(4), 322–334. https://doi.org/10.1177/0034355221993553. (PMID: 10.1177/0034355221993553) ; Lee, J., & Cagle, J. G. (2017). Validating the 11-item revised University of California Los Angeles scale to assess loneliness among older adults: An evaluation of factor structure and other measurement properties. The American Journal of Geriatric Psychiatry, 25(11), 1173–1183. https://doi.org/10.1016/j.jagp.2017.06.004. (PMID: 10.1016/j.jagp.2017.06.00428864097) ; Li, C. H. (2016). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48(3), 936–949. https://doi.org/10.3758/s13428-015-0619-7. (PMID: 10.3758/s13428-015-0619-726174714) ; Lumley, T. (2013). Biglm: Bounded memory linear and generalized linear models. R package version 0.9-1. Retrieved February 23, 2020, from https://CRAN.R-project.org/package=biglm. ; Markland, D. (2007). The golden rule is that there are no golden rules: A commentary on Paul Barrett’s recommendations for reporting model fit in structural equation modelling. Personality and Individual Differences, 42(5), 851–858. https://doi.org/10.1016/j.paid.2006.09.023. (PMID: 10.1016/j.paid.2006.09.023) ; Marsh, H. W., Hau, K.-T., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling, 11(3), 320–341. https://doi.org/10.1207/s15328007sem1103_2. (PMID: 10.1207/s15328007sem1103_2) ; Maydeu-Olivares, A., Shi, D., & Rosseel, Y. (2018). Assessing fit in structural equation models: A Monte-Carlo evaluation of RMSEA versus SRMR confidence intervals and tests of close fit. Structural Equation Modeling, 25(3), 389–402. https://doi.org/10.1080/10705511.2017.1389611. (PMID: 10.1080/10705511.2017.1389611) ; McNeish, D., & Wolf, M. G. (2023a). Dynamic fit index cutoffs for confirmatory factor analysis models. Psychological Methods, 28(1), 61–88. https://doi.org/10.1037/met0000425. (PMID: 10.1037/met000042534694832) ; McNeish, D., & Wolf, M. G. (2023b). Dynamic fit index cutoffs for one-factor models. Behavior Research Methods, 55(3), 1157–1174. https://doi.org/10.3758/s13428-022-01847-y. (PMID: 10.3758/s13428-022-01847-y35585278) ; McNeish, D., An, J., & Hancock, G. R. (2018). The thorny relation between measurement quality and fit index cutoffs in latent variable models. Journal of Personality Assessment, 100(1), 43–52. https://doi.org/10.1080/00223891.2017.1281286. (PMID: 10.1080/00223891.2017.128128628631976) ; Miller, A. J. (1992). Algorithm AS 274: Least squares routines to supplement those of Gentleman. Journal of the Royal Statistical Society, Series C (Applied Statistics), 41(2), 458–478. https://doi.org/10.2307/2347583. (PMID: 10.2307/2347583) ; Millsap, R. E. (2007). Structural equation modeling made difficult. Personality and Individual Differences, 42(5), 875–881. https://doi.org/10.1016/j.paid.2006.09.021. (PMID: 10.1016/j.paid.2006.09.021) ; Millsap, R. E. (2011). Statistical approaches to measurement invariance. Routledge. ; Millsap, R. E. (2013). A simulation paradigm for evaluating model fit. In M. Edwards & R. C. MacCallum (Eds.), Current topics in the theory and application of latent variable models (pp. 165–182). Routledge. ; Moshagen, M. (2012). The model size effect in SEM: Inflated goodness-of-fit statistics are due to the size of the covariance matrix. Structural Equation Modeling, 19(1), 86–98. https://doi.org/10.1080/10705511.2012.634724. (PMID: 10.1080/10705511.2012.634724) ; Moshagen, M., & Auerswald, M. (2018). On congruence and incongruence of measures of fit in structural equation modeling. Psychological Methods, 23(2), 318–336. https://doi.org/10.1037/met0000122. (PMID: 10.1037/met000012228301200) ; Moshagen, M., & Erdfelder, E. (2016). A new strategy for testing structural equation models. Structural Equation Modeling, 23(1), 54–60. https://doi.org/10.1080/10705511.2014.950896. (PMID: 10.1080/10705511.2014.950896) ; Muthén, B. (1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49(1), 115–132. https://doi.org/10.1007/BF02294210. (PMID: 10.1007/BF02294210) ; Muthén, B., Asparouhov, T. (2002). Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus. Retrieved May 12, 2019, from  https://www.statmodel.com/download/webnotes/CatMGLong.pdf. ; Muthén, L. K., Muthén, B. O. (1998-2017). Mplus user’s guide (version 8). Retrieved January 29, 2021, from https://www.statmodel.com/download/usersguide/MplusUserGuideVer_8.pdf. ; Muthén, B., Du Toit, S. H. C., Spisic, D. (1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes. Retrieved October 27, 2018, from http://www.statmodel.com/bmuthen/articles/Article_075.pdf. ; Niemand, T., & Mai, R. (2018). Flexible cutoff values for fit indices in the evaluation of structural equation models. Journal of the Academy of Marketing Science, 46, 1148–1172. https://doi.org/10.1007/s11747-018-0602-9. (PMID: 10.1007/s11747-018-0602-9) ; Nießen, D., Partsch, M. V., Kemper, C. J., & Rammstedt, B. (2019). An english-language adaptation of the social desirability–gamma short scale (KSE-G). Measurement Instruments for the Social Sciences, 2(1), 2. https://doi.org/10.1186/s42409-018-0005-1. (PMID: 10.1186/s42409-018-0005-1) ; Nye, C. D., & Drasgow, F. (2011). Assessing goodness of fit: Simple rules of thumb simply do not work. Organizational Research Methods, 14(3), 548–570. https://doi.org/10.1177/1094428110368562. (PMID: 10.1177/1094428110368562) ; OECD. (2013). OECD skills outlook 2013: First results from the survey of adult skills. OECD Publishing. https://doi.org/10.1787/9789264204256-en. (PMID: 10.1787/9789264204256-en) ; Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903. https://doi.org/10.1037/0021-9010.88.5.879. (PMID: 10.1037/0021-9010.88.5.87914516251) ; Pornprasertmanit, S., Wu, W., & Little, T. D. (2013). A Monte Carlo approach for nested model comparisons in structural equation modeling. In R. E. Millsap, L. A. van der Ark, D. M. Bolt, & C. M. Woods (Eds.), New developments in quantitative psychology (pp. 187–197). Springer. https://doi.org/10.1007/978-1-4614-9348-8_12. (PMID: 10.1007/978-1-4614-9348-8_12) ; Pornprasertmanit, S. (2014). The unified approach for model evaluation in structural equation modeling [Unpublished doctoral dissertation]. University of Kansas. Retrieved August 31, 2021, from http://hdl.handle.net/1808/16828. ; R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Retrieved March 15, 2020, from  http://www.r-project.org/index.html. ; Rammstedt, B., & Beierlein, C. (2014). Can’t we make it any shorter? The limits of personality assessment and way to overcome them. Journal of Individual Differences, 35(4), 212–220. https://doi.org/10.1027/1614-0001/a000141. (PMID: 10.1027/1614-0001/a000141) ; Reußner, M. (2019). Die Güte der Gütemaße: Zur Bewertung von Strukturgleichungsmodellen [The fit of fit indices: The evaluation of model fit for structural equation models]. Walter de Gruyter. ; Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354–373. https://doi.org/10.1037/a0029315. (PMID: 10.1037/a002931522799625) ; Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. Retrieved November 15, 2020, from  http://www.jstatsoft.org/v48/i02/ . ; Rutkowski, L., & Svetina, D. (2014). Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement, 74(1), 31–57. https://doi.org/10.1177/0013164413498257. (PMID: 10.1177/0013164413498257) ; Savalei, V. (2012). The relationship between root mean square error of approximation and model misspecification in confirmatory factor analysis models. Educational and Psychological Measurement, 72(6), 910–932. https://doi.org/10.1177/0013164412452564. (PMID: 10.1177/0013164412452564) ; Savalei, V. (2020). Improving fit indices in structural equation modeling with categorical data. Multivariate Behavioral Research, 56(3), 390–407. https://doi.org/10.1080/00273171.2020.1717922. (PMID: 10.1080/00273171.2020.171792232054327) ; Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23–74. ; Sharma, S., Mukherjee, S., Kumar, A., & Dillon, W. R. (2005). A simulation study to investigate the use of cutoff values for assessing model fit in covariance structure models. Journal of Business Research, 58(7), 935–943. https://doi.org/10.1016/j.jbusres.2003.10.007. (PMID: 10.1016/j.jbusres.2003.10.007) ; Shi, D., & Maydeu-Olivares, A. (2020). The effect of estimation methods on SEM fit indices. Educational and Psychological Measurement, 80(3), 421–445. https://doi.org/10.1177/0013164419885164. (PMID: 10.1177/001316441988516432425213) ; Shi, D., DiStefano, C., McDaniel, H. L., & Jiang, Z. (2018a). Examining chi-square test statistics under conditions of large model size and ordinal data. Structural Equation Modeling, 25(6), 924–945. https://doi.org/10.1080/10705511.2018.1449653. (PMID: 10.1080/10705511.2018.1449653) ; Shi, D., Maydeu-Olivares, A., & DiStefano, C. (2018b). The relationship between the standardized root mean square residual and model misspecification in factor analysis models. Multivariate Behavioral Research, 53(5), 676–694. https://doi.org/10.1080/00273171.2018.1476221. (PMID: 10.1080/00273171.2018.147622130596259) ; Shi, D., Lee, T., & Maydeu-Olivares, A. (2019). Understanding the model size effect on SEM fit indices. Educational and Psychological Measurement, 79(2), 310–334. https://doi.org/10.1177/0013164418783530. (PMID: 10.1177/001316441878353030911195) ; Simms, L. J., Zelazny, K., Williams, T. F., & Bernstein, L. (2019). Does the number of response options matter? Psychometric perspectives using personality questionnaire data. Psychological Assessment, 31(4), 557–566. https://doi.org/10.1037/pas0000648. (PMID: 10.1037/pas000064830869956) ; Soto, C. J., & John, O. P. (2017). The next Big Five Inventory (BFI-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. Journal of Personality and Social Psychology, 113(1), 117–143. https://doi.org/10.1037/pspp0000096. (PMID: 10.1037/pspp000009627055049) ; Steiger, J. H. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25(2), 173–180. https://doi.org/10.1207/s15327906mbr2502_4. (PMID: 10.1207/s15327906mbr2502_426794479) ; Ullman, J. B. (2014). Structural equation modeling. In B. G. Tabachnick & L. S. Fidell (Eds.), Using multivariate statistics (6th ed.). Pearson Education. ; Ushey, K. (2020). renv: Project environments. R package version 0.12.2. Retrieved November 5, 2020, from https://cran.r-project.org/web/packages/renv/index.html. ; Venables, W. N. & Ripley, B. D. (2002). Modern applied statistics with S-PLUS (4th ed.). Springer. ; Widaman, K. F., & Thompson, J. S. (2003). On specifying the null model for incremental fit indices in structural equation modeling. Psychological Methods, 8(1), 16–37. https://doi.org/10.1037/1082-989X.8.1.16. (PMID: 10.1037/1082-989X.8.1.1612741671) ; Xia, Y., & Yang, Y. (2018). The influence of number of categories and threshold values on fit indices in structural equation modeling with ordered categorical data. Multivariate Behavioral Research, 53(5), 731–755. https://doi.org/10.1080/00273171.2018.1480346. (PMID: 10.1080/00273171.2018.148034630477318) ; Xia, Y., & Yang, Y. (2019). RMSEA, CFI, and TLI in structural equation modeling with ordered categorical data: The story they tell depends on the estimation methods. Behavior Research Methods, 51, 409–428. https://doi.org/10.3758/s13428-018-1055-2. (PMID: 10.3758/s13428-018-1055-229869222) ; Yuan, K. H., & Bentler, P. M. (2000). 5. Three likelihood-based methods for mean and covariance structure analysis with nonnormal missing data. Sociological Methodology, 30(1), 165–200. https://doi.org/10.1111/0081-1750.00078. (PMID: 10.1111/0081-1750.00078)
  • Contributed Indexing: Keywords: Confirmatory factor analysis; Fit index; Goodness-of-fit; Ordered categorical data; Structural equation modeling
  • Entry Date(s): Date Created: 20230828 Date Completed: 20240529 Latest Revision: 20240531
  • Update Code: 20240531
  • PubMed Central ID: PMC11133148

Klicken Sie ein Format an und speichern Sie dann die Daten oder geben Sie eine Empfänger-Adresse ein und lassen Sie sich per Email zusenden.

oder
oder

Wählen Sie das für Sie passende Zitationsformat und kopieren Sie es dann in die Zwischenablage, lassen es sich per Mail zusenden oder speichern es als PDF-Datei.

oder
oder

Bitte prüfen Sie, ob die Zitation formal korrekt ist, bevor Sie sie in einer Arbeit verwenden. Benutzen Sie gegebenenfalls den "Exportieren"-Dialog, wenn Sie ein Literaturverwaltungsprogramm verwenden und die Zitat-Angaben selbst formatieren wollen.

xs 0 - 576
sm 576 - 768
md 768 - 992
lg 992 - 1200
xl 1200 - 1366
xxl 1366 -