Exploratory structural equation modeling, bifactor models, and standard confirmatory factor analysis models: Application to the BASC-2 Behavioral and Emotional Screening System Teacher Form

https://doi.org/10.1016/j.jsp.2013.09.001Get rights and content

Abstract

Several psychological assessment instruments are based on the assumption of a general construct that is composed of multiple interrelated domains. Standard confirmatory factor analysis is often not well suited for examining the factor structure of such scales. This study used data from 1885 elementary school students (mean age = 8.77 years, SD = 1.47 years) to examine the factor structure of the Behavioral Assessment System for Children, Second Edition (BASC-2) Behavioral and Emotional Screening System (BESS) Teacher Form that was designed to assess general risk for emotional/behavioral difficulty among children. The modeling sequence included the relatively new exploratory structural equation modeling (ESEM) approach and bifactor models in addition to more standard techniques. Findings revealed that the factor structure of the BASC-2 BESS Teacher Form is multidimensional. Both ESEM and bifactor models showed good fit to the data. Bifactor models were preferred on conceptual grounds. Findings illuminate the hypothesis-generating power of ESEM and suggest that it might not be optimal for instruments designed to assess a predominant general factor underlying the data.

Introduction

Several theories and assessment instruments in psychology and related disciplines are based on the assumption that a general construct is composed of several closely related domains. As noted by Reise, Morizot, and Hays (2007), such broadly defined underlying constructs have content heterogeneous indicators. Prominent examples of such theories and assessment instruments include research on health-related quality of life (e.g., Lehman, 1988, Stewart and Ware, 1992), the Big Five personality factors (e.g., Costa and McCrae, 1992, Costa and McCrae, 1995, McCrae and Costa, 2004), achievement (e.g., Jackson, Ahmed, & Heapy, 1976), intelligence and cognitive abilities (e.g., Gustafsson and Balke, 1993, Spearman, 1904, Spearman, 1927), and psychopathy in adulthood (e.g., Hare, 2003, Patrick et al., 2007). In most of these cases, the central scientific or clinical interest lies in the hypothesized general construct.

The Behavioral and Emotional Screening System (BESS; Kamphaus & Reynolds, 2007), which is part of the Behavioral Assessment System for Children, Second Edition (BASC-2) family of instruments (Reynolds & Kamphaus, 2004), is also built upon the assumption of a general construct that is composed of several closely related domains. The BASC-2 BESS was developed to provide brief assessment instruments that can be used to screen school-age students for emotional and behavioral problems. The BASC-2 BESS Teacher Form, which is the focus of the current study, was developed by selecting items from several larger item pools that targeted four interrelated dimensions: low adaptive skills, externalizing problems, internalizing problems, and school problems (Reynolds & Kamphaus, 2004). The BASC-2 BESS Teacher Form is thought to measure one general construct (i.e., a broad at-risk factor of behavioral, emotional, and academic problems) characterized as the “behavioral and emotional strengths and weaknesses of children” (Kamphaus & Reynolds, 2007, p. 1). The resulting total composite score of the BASC-2 BESS Teacher Form can be used to identify students at-risk for behavioral and emotional problems in need of comprehensive evaluations and interventions.2

Within the latent variable analysis framework, several approaches, such as bifactor models, higher-order factor models, and exploratory structural equation modeling (ESEM), are available that might be particularly well-suited to examining the underlying factor structure of assessment instruments that are predicated on the assumption of general constructs composed of several closely related domains. Some of these approaches—especially ESEM (Asparouhov & Muthén, 2009)—have been introduced to the literature only recently, and comparison studies are consequently rare. Therefore, it is of high interest to the field to incorporate these innovative approaches into a model testing series with a real-life data set to improve our understanding of the strengths and potential boundaries of each approach. The purpose of this study was to contribute to this literature by applying (a) standard confirmatory factor analysis (CFA), (b) bifactor modeling, (c) higher-order factor modeling, and (d) ESEM approaches to investigate the factor structure of a relatively recently developed screening instrument for assessing behavioral and emotional problems of school-age children (i.e., the BASC-2 BESS Teacher Form).

Approaches to the specification of latent variable measurement models in psychology and related fields have, over the last two or three decades, been predominated by standard confirmatory factor analysis (CFA), in which several zero factor loading restrictions are imposed (often adopting a simple structure specification) to represent the hypothesis that only specific latent factors influence specific manifest indicators. However, the restrictive measurement model approach of standard CFA is often not well aligned to the analysis of assessment instruments composed of indicators with many cross-loadings (Asparouhov & Muthén, 2009). This has likely contributed to various questionable practices in much applied CFA research (see Asparouhov and Muthén, 2009, Browne, 2001, Marsh et al., 2009).

Within the confirmatory factor analysis framework, two alternative (and less restrictive) modeling approaches–bifactor models and higher-order factor models–are available that might be better suited for investigating the factor structure of instruments that are composed of indicators with many cross-loadings. Bifactor models, also referred to as generalspecific models or nested models, include a general factor posited to account for the commonality of all manifest variables and several orthogonal (i.e., uncorrelated) specific factors representing the hypothesized unique influence of the specific factors on subsets of the manifest variables above and beyond the effects of the general factor. In other words, each manifest variable is a reflective indicator of both a general factor and a more narrowly defined specific factor that is not correlated with the general factor. Thus, the variance of each manifest variable is decomposed into a larger number of separate components compared to what is done in standard CFA models. A discussion of various bifactor modeling strategies can be found elsewhere (e.g., Chen, West, & Sousa, 2006). Bifactor models have been used in research on intelligence (e.g., Gustafsson & Balke, 1993), various psychopathy constructs for youths and adults (e.g., Kimonis et al., 2008, Patrick et al., 2007), and health-related outcomes (e.g., Chen et al., 2006, Reise et al., 2007).

Higher-order factor models, in contrast, posit that more specialized facets of a construct are influenced by a broader dimension of the general construct of interest. In other words, a higher-order factor is hypothesized to account for the covariation among multiple lower-order factors. A more thorough overview of higher-order factor models is offered elsewhere (e.g., Brown, 2006, Chen et al., 2006, Rindskopf and Rose, 1988). They are more widely known in psychology-related fields and have been applied to a range of areas, including anxiety sensitivity (e.g., Zinbarg, Barlow, & Brown, 1997), cognitive abilities (e.g., Neuman et al., 2000, Reynolds et al., 2013), self-concept (e.g., Marsh, Ellis, & Craven, 2002), and personality (e.g., Judge, Erez, Bono, & Thoresen, 2002). It has been argued that bifactor models have several potential advantages over higher-order models (Chen et al., 2006), especially when the focal interest lies in examinations of the predictive relations between specific factors and several external criteria, above and beyond the general higher-order factor.

A different modeling strategy predicated on the integration of confirmatory and exploratory factor analysis has been recently introduced to the literature in the form of ESEM (Asparouhov & Muthén, 2009) and applied to the Big Five personality factors (Marsh et al., 2010) and students' evaluations of university teaching (Marsh et al., 2009). The ESEM approach differs from standard CFA in that all factor loadings are estimated, observing various constraints necessary for model identification, and factor loading matrices can be rotated. More details about ESEM are provided in Asparouhov and Muthén (2009) and Marsh et al. (2009). It has been argued that ESEM is a viable alternative to standard CFA for psychological scales composed of indicators with many nonzero cross-loadings (Marsh et al., 2009), but few comparison studies for real data are available because the approach was implemented only recently in a major statistical software package. This issue clearly merits further research.

The primary aim of this study was to incorporate newer latent measurement modeling approaches (particularly ESEM) into an integrative model testing series of a real-data assessment instrument to contribute to a richer understanding of the strengths and potential boundaries of alternative approaches in the analysis of instruments composed of indicators sampled from several interrelated domains. A secondary and substantive aim was to examine the factor structure of a screening instrument assessing behavioral and emotional problems of school-age children, the BASC-2 BESS Teacher Form (Kamphaus & Reynolds, 2007). As described earlier, the BASC-2 BESS Teacher Form was developed by selecting a subset of items from larger item pools that assessed students on four interrelated dimensions posited to jointly reflect a single general construct of “at-riskness” for emotional and behavioral difficulties. Exploratory principal component analysis was conducted separately for each of the four dimensions to identify items with the highest loadings for subsequent inclusion in the BASC-2 BESS Teacher Form (Kamphaus and Reynolds, 2007, Reynolds and Kamphaus, 2004). It has been pointed out by others that this “two-stage sampling procedure (i.e., domains within a construct and items within domains) frequently produces rating scales with a multidimensional structure” (Gibbons et al., 2007, p. 4). No data are provided in the BASC-2 BESS manual regarding the unidimensionality of the instrument's factor structure (Kamphaus & Reynolds, 2007). DiStefano and Morgan (2010) found mediocre support for unidimensionality of the BASC-2 BESS Teacher Form in their item response theory analysis and called for systematic investigation of the dimensional structure of the instrument. We are unaware of any other published research that tested the factor structure of the BASC-2 BESS Teacher Form.3 Thus, this study aim addresses an important substantive issue of relevance to applied practitioners in the mental health and school-related professions in utilizing technically adequate universal screenings to identify those in need of services.

On the basis of the discussed methodological and substantive considerations, the following model specifications were tested for the BASC-2 BESS Teacher Form. First, we investigated a standard unidimensional CFA model in which the covariation among all 27 items was explained by a single first-order factor. This model arguably resembled the intended dimensional structure of the screener most closely (see Kamphaus & Reynolds, 2007). Second, an ESEM model with four correlated factors was tested that included a priori target loadings of items on one of the four factors (i.e., Externalizing Problems, Internalizing Problems, School Problems, and Low Adaptive Skills), while allowing for cross-loadings on the other three factors. Third, a standard four-factor CFA model (see Fig. 1) was estimated in which all cross-loadings were fixed to the value zero, with four correlated factors (i.e., Externalizing Problems, Internalizing Problems, School Problems, and Low Adaptive Skills) that each accounted for the covariation among their respective subset of items. Thus, an important difference between the specified standard four-factor CFA model and the ESEM model with four correlated factors is that the former did not permit cross-loadings of items on multiple factors. Fourth, a bifactor model (see Fig. 2) was estimated in which there was a general factor of Maladaptive Behavior (on which all 27 items loaded) and four specific factors of Externalizing Problems, Internalizing Problems, School Problems, and Low Adaptive Skills (on which only their respective subsets of items loaded), with all factors being specified as uncorrelated with each other. Thus, the bifactor model differs from the standard four-factor CFA model in that, under the bifactor model, all items load on their respective specific factor as well as on the general factor and that all factors are uncorrelated. Finally, a higher-order factor model (see Fig. 3) was estimated, with Externalizing Problems, Internalizing Problems, School Problems, and Low Adaptive Skills specified as the four first-order factors and Global Risk for Emotional and Behavior Problems specified as a second-order factor. Thus, the higher-order factor model differs from the bifactor model in that the four specific factors are not assumed to be uncorrelated and distinct but their intercorrelations are explained by the second-order general factor.

Section snippets

Procedure and sample

Approval to conduct this study was obtained from the authors' institution and from a suburban school district in the South. Demographic and screening data were provided by the school district in de-identified format to the authors. All four elementary schools within the district participated in the school screening. The screening was conducted in October of 2010 near the beginning of the school year. All teachers were asked to complete the BASC-2 BESS Teacher Form for each student as part of

Descriptive information

Fairly strong floor effects (i.e., with over 90% of the cell counts being concentrated in the lower two response categories of the Likert-scale) were observed for 9 of the 27 items of the BASC-2 BESS Teacher Form: items 3, 11, 12, 13, 14, 15, 20, 22, and 25. There were no empty cells, however, and it was not necessary to collapse response categories because of sparse data. Given the item content, floor effects are not surprising. Of the 9 items demonstrating floor effects, 6 items require the

Discussion

The purpose of this study was to examine the factor structure of the BASC-2 BESS Teacher Form (Kamphaus & Reynolds, 2007). This screener was designed to measure the general construct of risk for emotional and behavioral difficulty, posited to consist of four interrelated domains, among school-age children in the United States. We are not aware of other research that has systematically examined the underlying factor structure of the BASC-2 BESS Teacher Form (although recent factor-analytic work

Acknowledgment

This project was conducted with support from the Small Grant Program by the University of Houston. The authors wish to acknowledge all data assistants with the project: Kerri Nowell, Victoria Faulkner, Moureen Azagidi, Danielle Knight, Christie Brewton, and Denisse Coronado.

References (52)

  • P.M. Bentler et al.

    Significance tests and goodness of fit in the analysis of covariance structures

    Psychological Bulletin

    (1980)
  • R. Bernstein et al.

    An older and more diverse nation by midcentury

  • R. Bickel

    Multilevel analysis for applied research

    (2007)
  • T.A. Brown

    Confirmatory factor analysis for applied research

    (2006)
  • M.W. Browne

    An overview of analytic rotation in exploratory factor analysis

    Multivariate Behavioral Research

    (2001)
  • M.W. Browne et al.

    Alternative ways of assessing model fit

  • F.F. Chen et al.

    A comparison of bifactor and second-order models of quality of life

    Multivariate Behavioral Research

    (2006)
  • P.T. Costa et al.

    Revised NEO Personality Inventory (NEO-PI-R) and NEO Five-Factor Inventory (NEO-FFI): Professional Manual

    (1992)
  • P.T. Costa et al.

    Domains and facets: Hierarchical personality assessment using the Revised NEO Personality Inventory

    Journal of Personality Assessment

    (1995)
  • C. DiStefano et al.

    Evaluation of the BESS TRS-CA using the Rasch rating scale model

    School Psychology Quarterly

    (2010)
  • E. Dowdy et al.

    Factor structure of the BASC-2 Behavioral and Emotional Screening System Student Form

    Psychological Assessment

    (2011)
  • S.J. Finney et al.

    Non-normal and categorical data in structural equation modeling

  • D.B. Flora et al.

    An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data

    Psychological Methods

    (2004)
  • R.D. Gibbons et al.

    Full-information item bifactor analysis of graded response data

    Applied Psychological Measurement

    (2007)
  • J.-E. Gustafsson et al.

    General and specific abilities as predictors of school achievement

    Multivariate Behavioral Research

    (1993)
  • R.D. Hare

    The Hare Psychopathy Checklist-Revised 2

    (2003)
  • Cited by (0)

    1

    Tel.: + 1 281 456 6858; fax: + 1 281 727 2094.

    View full text