| Sign In to gain access to subscriptions and/or personal tools. |
Matrix Summaries Improve Research Reports: Secondary Analyses Using Published Literature
LINDA REICHWEIN ZIENTEK is an assistant professor of mathematics education at Sam Houston State University, Department of Mathematics and Statistics, P.O. Box 2206, Huntsville, TX 77341; lrzientek{at}shsu.edu. Her research focuses on mathematics teacher preparation, teacher induction programs, and quantitative research methods.
Correlation matrices and standard deviations are the building blocks of many of the commonly conducted analyses in published research, and AERA and APA reporting standards recommend their inclusion when reporting research results. The authors argue that the inclusion of correlation/covariance matrices, standard deviations, and means can enhance findings in education and psychology by permitting secondary researchers to (a) conduct commonly utilized traditional univariate and multivariate analyses not initially performed in primary studies, (b) produce effect sizes and other statistics not included in prior published literature, and (c) conduct analyses once difficult to perform. Furthermore, meta-analytic thinking is encouraged when researchers have the ability to conduct the same analyses on multiple studies and then compare these findings across studies.
Key Words: correlation • general linear model • reporting standards • secondary data analyses The 2006 American Educational Research Association (AERA) Standards for Reporting on Empirical Social Science Research in AERA Publications required that in quantitative studies "relevant descriptive statistics (such as means and standard deviations for continuous variables, frequencies for discrete variables with few categories, and correlation matrices)" (p. 36) should be reported or made available by authors. The 2001 Publication Manual of the American Psychological Association (APA) noted that in research reports, "for correlational analyses (e.g., multiple regression analysis, factor analysis, and structural equation modeling), the sample size and variance-covariance (or correlation) matrix are needed" (p. 23). Thompson (2008) reviewed the etiology of these and related publication standards. Because all analyses (e.g., t tests, ANOVAs, MANOVAs) are correlational (see Bagozzi, Fornell, & Larcker, 1981; Cohen, 1968; Knapp, 1978), the necessity of reporting matrix summaries is not limited to the three specific analyses provided in APA’s noninclusive list. Scholars benefit greatly when matrix summaries (e.g., correlation matrix and standard deviations, or the variance-covariance matrix) are made available for continuous data.
Matrix Summaries The correlation matrix is a symmetric matrix whose off-diagonal entries are the Pearson product-correlation coefficients (rXY). Because the bivariate correlation between a variable and itself is always 1, the diagonal entries of the correlation matrix are all 1s. When reporting correlation matrices, the APA (2001) Publication Manual recommended entering dashes in place of the 1s along the diagonal. In the case where the correlation matrix is not sufficient for further secondary analyses, and the variance-covariance matrix is needed, the variance-covariance matrix can be computed given (a) the correlation matrix and (b) standard deviations. The variance-covariance matrix is a symmetric matrix whose diagonal entries are the variances, and off-diagonal entries are the covariances between X and Y (COVXY). The COVXY equals rXY x SDX x SDY. The APA Publication Manual recommended reporting only the upper or the lower portion of correlation and variance-covariance matrices, because the two off-diagonal triangles of symmetric matrices are redundant. The correlation matrix serves as the foundation for all analytic methods, as suggested by the concept called the general linear model (GLM).
The General Linear Model The vast majority of parametric statistical procedures in common use are part of [a single analytic family called] the General Linear Model (GLM), including the t test, analysis of variance (ANOVA), multiple regression, descriptive discriminant analysis (DDA), multivariate analysis of variance (MANOVA), canonical correlation analysis (CCA), and structural equation modeling (SEM). Moreover, these procedures are hierarchical [italics added], in that some procedures are special cases of others. (p. 485) In 1968, Cohen argued that multiple regression analysis subsumes all univariate parametric statistical analyses as special cases. In 1978, Knapp showed that all commonly utilized univariate and multivariate analyses are special cases of canonical correlation analysis (CCA). In 1981, Bagozzi and colleagues demonstrated that structural equation modeling (SEM) is an even more general case of the GLM (see Fan, 1997, for a cogent explanation). Figure 1 presents a conceptual map of the commonly used statistical analyses falling within GLM. Note that predictive discriminant analysis, unlike descriptive discriminant analysis (DDA), is not part of the GLM (Huberty, 1994). It can also be shown that the mathematics of factor analysis are used to compute the multiplicative weights applied to the measured variables, either explicitly or implicitly, in all analyses throughout the GLM.
As noted by Thompson (1984), "The first step in a canonical correlation analysis involves the calculation of the intervariable correlation matrix" (p. 11), which is precisely why the majority of the commonly utilized analyses within the GLM can be conducted using only matrix summaries. In 1996, Campbell and Taylor demonstrated Knapp’s conclusion (i.e., CCA subsumes both univariate and multivariate parametric analyses) by conducting all common quantitative analyses with a CCA computer program syntax. The necessity of the correlation matrix for conducting CCA, in conjunction with evidence that CCA subsumes the most commonly utilized univariate and multivariate parametric analyses (e.g., t tests, ANOVA, MANOVA, DDA), logically confirms that both univariate and multivariate parametric analyses are correlational in nature.
Purpose We hope our few examples illustrate how "meta-analytic thinking" (cf. Cumming & Finch, 2001; Fidler & Thompson, 2001; Thompson, 2006b) can be encouraged when researchers have the ability to conduct the same analyses on multiple studies and then compare these findings across studies and over time. Thompson (2002) defined meta-analytic thinking as both (a) the prospective formulation of study expectations and design by explicitly invoking prior effect sizes and (b) the retrospective interpretation of new results, once they are in hand, via explicit, direct [italics added] comparison with the prior effect sizes in the related literature. (p. 28)Our examples may also facilitate broader recognition that all parametric statistical analyses are correlational in nature.
Syntax Commands and Our Reporting of Decimal Places We present our results here to more decimal points than many researchers use, to allow readers to replicate our secondary analysis examples with more precision. We also believe reporting numerical results to more decimal places in primary reports might facilitate more precise secondary analyses. Historically, the potential contributions of secondary analyses may have been underestimated within the field.
Reporting matrix summaries provides researchers the necessary resources to answer different questions and conduct different analyses than those performed in initial research reports. As advances in technology are made, researchers can locate archived studies and conduct analyses once considered difficult or impossible to conduct.
Univariate Analysis When matrix summaries are provided, secondary researchers can (a) conduct univariate analyses such as ANOVA, multiple regression, and path analysis; (b) produce information lacking in prior published literature; and (c) conduct analyses once considered difficult to conduct, such as commonality analyses (Nimon, Lewis, Kane, & Haynes, 2008; Zientek & Thompson, 2006). This is important because despite the widespread use of ANOVA and regressions, effect sizes have not historically been reported for ANOVA (cf. Kieffer et al., 2001; Snyder & Thompson, 1998; Thompson & Snyder, 1998), and researchers have been inconsistent in the reporting of effect sizes, inferential statistics, and p values—particularly when results are not statistically significant (Zientek et al., 2008). Both AERA (2006) and APA (2001) recommended the reporting of inferential statistics, p values, and effect sizes. Reporting matrix summaries allows secondary researchers to compute and interpret all relevant findings, even those not published in the original study.
Multivariate Analyses
Furthermore, univariate and multivariate analyses even of the same data (a) address different research questions and (b) analyze different variables. For example, in a single study looking at gender differences across two outcome variables, X and Y, two univariate ANOVAs might yield two pCALCULATED values both of .77 and two univariate This is why night-and-day different results can be obtained when using multivariate as opposed to univariate analyses of exactly the same data (Fish, 1988). This is also why the use of ANOVAs post hoc to find statistically significant MANOVA effects is illogical, given that the two analyses address different research questions and also focus on different variables. Notwithstanding the fact that about three quarters of published MANOVA articles incorrectly report post hoc ANOVAs (Kieffer et al., 2001), "one would usually be well advised to follow up a significant multivariate test with a [descriptive] discriminant function analysis in order to study the nature of the group differences more closely" (Tatsuoka, 1971, p. 50).
Multivariate Analysis of Variance
Score Validity: Factor Analysis In a review of quantitative reporting practices in teacher education research, Zientek et al. (2008) found that relatively few (6%) of the 174 reviewed studies conducted EFAs. When EFAs were conducted, the majority of the studies did not report the rotation method or both pattern and structure coefficients (Zientek et al., 2008), even when oblique rotations were performed, and then these matrices differ (Thompson, 2004). Henson and Williams (2006) found that faculty members at 38.5% of education doctoral programs believed that few or none (i.e., less than 25%) of their doctoral graduates were "capable of applying the technique or concept [factor analysis] independently in their own research" (p. 30). Advances in technology have increased the ease in which EFAs can be conducted (Henson & Roberts, 2006). However, EFAs require a number of decisions in determining the number of factors to extract and the rotation methods to be used. In published research where authors conducted EFAs but did not report rotation methods, pattern/structure coefficients, or scree plots, the secondary researcher can use the published matrix summaries to reproduce full EFA results and thus evaluate reported results based on complete information and correct analyses (Thompson, 2004).
Structural Equation Modeling When conducting an SEM, there is the occasional case when using the raw data set is necessary. According to Kline (2005), when non-normal data are being analyzed or special estimation methods are being used that do "not assume normal distributions or accommodate cases with missing observations," matrix summaries may not be appropriate (p. 46). However, for the majority of all other cases, inputting matrix summaries is appropriate and indeed may offer advantages to conducting the analysis with the raw data set. According to Kline, these advantages include the following: (a) When sample sizes are large, the statistical program is only required to work with a data file that is as long as the number of variables; (b) conducting analyses with matrix summaries allows readers to "either replicate the original analyses or estimate alternative models not considered in the original work" (p. 46); and (c) models can be formulated based on theory or meta-analysis and be submitted for secondary exploratory or diagnostic purposes.
Encourage Meta-Analytic Thinking
Published articles were utilized to illustrate how reporting matrix summaries (i.e., standard deviations and correlation matrix, or variance-covariance matrix), means, and sample size can be used in secondary research using commonly available software. Space precludes illustration of all possible secondary analyses or the pursuit of all possible secondary research purposes. The criteria for article selection were (a) matrix summaries, means, and sample sizes for the variables of interest were reported; (b) the new analyses were logical for the variables in the study; and (c) the article was archived in a peer-reviewed education or psychology journal or in ERIC. Because the most commonly utilized univariate and multivariate parametric analyses are all correlational in nature, our examples were not limited to multiple regressions, factor analyses, and SEMs. The statistical analyses were conducted in SPSS by inputting matrix summaries, means, and sample sizes. The syntax commands we present here may be modified by readers to conduct similar secondary analyses with other research reports.
MANOVA, CCA, and EFA were conducted as secondary analyses for selected prior studies. In addition, commonality analysis as a supplementary secondary analysis for multiple regression was performed.
Multivariate Analysis of Variance Wilkins (2008) "investigated 481 in-service elementary teachers’ levels of mathematical content knowledge, attitudes toward mathematics, beliefs about the effectiveness of inquiry-based instruction, [and] use of inquiry-based instruction" (p. 139). Among other analyses, Wilkins ran four separate ANOVAs to test mean differences between primary (Grades K–2) and upper elementary (Grades 3–5) teachers, with respect to content knowledge, mathematics attitudes, instructional beliefs, and instructional practices. In Wilkins’ (2008) study, many of these variables was correlated at a noteworthy level, and the items shared a common conceptual meaning. In the present analyses, MANOVA was employed to investigate mean differences between primary (Grades K–2) and upper elementary (Grades 3–5) teachers on four variables. Wilkins provided the recommended means and matrix summaries within the primary report (p. 149). The SPSS syntax for conducting a MANOVA in secondary analysis is presented in Figure 2. The interested reader is encouraged to consult the original report and then retype and run our syntax to see how to conduct such secondary analyses.
The secondary MANOVA results indicated that statistically significant differences existed between the primary (Grades K–2) and upper elementary (Grades 3–5) teachers across the four response variables, F(4/476) = 17.16, p < .001, with a noteworthy effect size (  2 = 12.6%). The effect size was computed as 1 – Wilks’ lambda (Stevens, 2002). In contrast with Wilkins’ ANOVA results, the secondary analysis revealed that the largest of the four (unreported) univariate 2 values (i.e., ANOVA 2 = 7.4%) was only roughly half our multivariate effect size. Thus our secondary analysis also enabled the computation of univariate effect sizes not reported in the primary report. As emphasized previously, it is theoretically conceivable for ANOVA and MANOVA analyses of the same data to yield results that are night-and-day different (i.e., respectively, have near 1.0 p values and near-zero effect sizes versus near-zero p values and near 1.0 variance-accounted-for effect sizes; see Fish, 1988). In such cases, only the multivariate analyses take into account all possible simultaneous relationships among the variables and thus honor the ecological reality that the variables in reality coexist.
Canonical Correlation Analysis
Here we conducted a CCA in secondary analysis of the data reported by Wentzel (2002, p. 294). In the primary report, Wentzel conducted four separate hierarchical multiple regressions to determine if "the five teaching dimensions [Fairness, Teacher Motivation, Rule Setting, Negative Feedback, High Expectations] explained significant amounts of variance in student motivation, social behavior, and achievement [Prosocial Pursuit, Responsibility Pursuit, Class Interest, Mastery Orientation]" (p. 287). Conducting multiple tests increases the likelihood of experiment-wise Type I errors and fails to honor a reality in which the variables simultaneously coexist. Therefore, we conducted a CCA (see Thompson, 1984, 2000a) between the variable sets of (a) five teacher beliefs/behaviors and (b) four student goals and interests outcomes.
CCA produces orthogonal (i.e., uncorrelated) functions (i.e., sets of multiplicative weights, like regression β weights or factor pattern coefficients) that optimize the multivariate relationships between the variable sets. In the more familiar regression analysis, multiplicative weights (often called β weights) are applied to the measured predictor variables to obtain scores on the latent predicted Y or In canonical analysis, sets of multiplicative weights are applied to the predictor variables to obtain scores on unobserved latent predictor variables, and sets of multiplicative weights are also applied to the criterion or outcome variables to obtain scores on unobserved latent outcome variables. Indeed, the Pearson r2 values between the latent predictor and the latent outcome variables are the multivariate squared canonical correlations (i.e., the RC2 values). A more complete explanation of this weighting process using manageable examples involving only 10 or so people is provided by Thompson (1984, 1991). The number of uncorrelated canonical functions equals the number of variables in the smaller variable set (i.e., in our secondary analysis, four). However, the squared canonical correlation coefficients (i.e., respectively, 44.0%, 4.5%, 0.8%, and 0.5%) from our secondary analyses suggested that only the first two functions were noteworthy. The canonical coefficients from our secondary analysis are reported in Table 1.
Our results were consistent with Wentzel’s finding that High Expectations was a positive predictor of motivation outcomes. However, when comparing our findings with the four multiple regression analyses originally reported, the contributions of Teacher Motivation and Fairness were underestimated in the primary regression analyses. In the CCA results, Teacher Motivation and Fairness each accounted for approximately 44% of the variance in Function I, even though High Expectations also contributed 65.1% of its variance to Function I. And our findings indicate that when all four motivation outcomes were simultaneously entered in a single model, Class Interest shared almost all its variance (89.7%) with Function I and thus explained most of the multivariate relationship between the two sets of latent variables.
Factor Analysis EFAs require a logical sequence of steps. For example, the number of factors to extract and the rotation method must be determined. Guttman’s (1954) eigenvalue-greater-than-1 rule is often used to determine the number of factors, but this method may lead to extraction of too many factors (Gorsuch, 1983; Thompson 2004). Alternatively, the scree test (Cattell, 1966) determines the number of factors as the point where the graph of the eigenvalues across the unrotated factors tapers off. Advantages to conducting an EFA from published studies, even when researchers themselves conducted an EFA, include the ability to (a) base factor extraction decisions on scree plot or parallel analysis results, even if authors based their decisions on Guttman’s rule, and (b) conduct an EFA with a different rotation method when theory supports alternate models (e.g., correlated factors). To illustrate, an EFA with a varimax rotation was conducted to investigate the nature of the constructs in the Steinmayr and Spinath (2009) study, which investigated the role of motivation and ability in predicting student achievement, by using a combination of well-known instruments. In the primary study, a number of regression analyses were performed to predict student achievement, even though the authors noted (p. 5) substantial multicollinearity among some of the predictors. Steinmayr and Spinath (2009) did not conduct a factor analysis of the predictors, notwithstanding the noted multicollinearity. However, sufficient statistics for an EFA were reported (p. 5). The scree plot and Guttman rule results were in agreement on the number of factors to extract. The varimax-rotated pattern/structure coefficients for the four factors are presented in Table 2.
Our analyses identified four interpretable uncorrelated factors, which we labeled Intelligence, Performance Preferences, Mathematics Versus Language Orientation, and Achievement Motivation. Our results suggest that a more parsimonious analysis could have been performed by using factor scores on these four constructs to predict academic success, rather than using the original numerous collinear predictors.
Commonality Analysis
With more than three predictor variables, commonality analyses have historically been difficult to conduct because as the number of predictor variables (k) increases, the number of coefficients (2k – 1) exponentially increases. However, the recent development of a free computer program by Nimon et al. (2008) facilitates the analysis of even complex problems. Commonality analysis can be employed to supplement either univariate (Mood, 1969; Zientek & Thompson, 2006) or multivariate (Frederick, 1999) analyses. The primary report by Pajares and Graham (1999, p. 130) provides the data for our secondary analysis. The researchers employed a number of variables to predict school performance in a series of regression analyses and did not conduct commonality analysis as a vehicle to explore all the unique and common explanatory abilities of the predictors. Here we use a subset of their predictors (i.e., Self-Efficacy, Anxiety, and Self-Concept) to illustrate secondary commonality analysis. This analysis quantifies the percentage of the sum-of-squares of the outcome variable associated uniquely with each of the three predictor variables (i.e., areas 1–3 in Figure 4), the percentage associated with each of the three combinations of the three predictors taken two at a time (i.e., areas 4–6 in Figure 4), and the percentage of area commonly explained by all three predictors (i.e., area 7 in Figure 4), as exemplified in Figure 4. Figure 4 is a generic graphic describing any commonality analysis involving exactly three predictor variables.
As explained by Thompson (2006a), the first step in a commonality analysis is to compute the R2 values for all combinations of the predictors. The SPSS syntax to obtain the required values in a secondary analysis is presented in Figure 5. For these data, the relevant values are:
Next, formulas (Thompson, 2006a, p. 279) are used to compute the predictive power unique only to a given predictor, and the predictive power common to the predictors in all their combinations. For these data the results are:
These seven partitions of the R2 of 34.4% using all three predictors are presented in Table 3. The results make clear that Self-Efficacy dominates the explanation of School Performance in the analysis. The Self-Efficacy variable alone explained r2SCHOOL PERFORMANCE x SELF-EFFICACY = 32.49% of the variance in School Performance, and of the total variance explained by all three predictors together (i.e., R2 = 34.40%), Self-Efficacy uniquely accounted for nearly a third of the explained variance (i.e., USELF-EFFICACY / R2 = 10.35% / 34.40% = 0.301).
Conducting a study is analogous to detective work. Given the same research questions, two different researchers may develop completely different study designs and conduct completely different analyses. When matrix summaries are provided in primary research, variables later can be investigated from different perspectives and new secondary analyses can be conducted—even after the passage of time as new statistical methods are developed. Secondary analysis also supports the maturation of the culture of respectful criticism so important in academic organizations. In their article in Educational Researcher, Feuer, Towne, and Shavelson (2002) asked, What are the most effective means of stimulating more and better scientific educational research? . . . The primary emphasis [italics added] should be on nurturing and reinforcing a scientific culture of educational research. . . . The development of a scientific culture rests with individual researchers, supported by leadership in their professional associations. (p. 4)They defined scientific culture as "a set of norms and practices and an ethos of honesty, openness, and continuous reflection, including how research quality is judged [italics added]" (Feuer et al., 2002, p. 4). Thompson (2008) argued that an organizational culture of respectful criticism is needed to support vibrant, mature scholarly dialogue. The present report illustrated the value of providing means, sample size, and matrix summaries (e.g., means, standard deviations, and the correlation matrix or variance-covariance matrix) for (a) reanalyzing data from different perspectives or for addressing different research questions, (b) producing information lacking in primary literature (e.g., effect sizes), and (c) conducting analyses once considered too difficult to conduct (e.g., SEM prior to the creation of "graphics in/graphics out" software interfaces for declaring models). These illustrations endorse the underlying realization that all common analyses are correlational. Our report also illustrated how to conduct secondary analyses by typing syntax for use in commonly available software. Correlation matrices and standard deviations—or variance-covariance matrices—are the building blocks of many of the analyses commonly conducted in published research (Cohen, 1968; Knapp, 1978). Because the first step in many of these analyses involves the use of the correlation or covariance matrix, the computer must first compute the correlation or covariance matrix. Providing the association coefficient matrix saves the computer one set of calculations but, more important, allows researchers to conduct a number of secondary analyses without the original data set. The illustrations in the present study suggest that reporting matrix summaries, means, and sample sizes can benefit research. In particular, reporting this information allows secondary researchers to investigate the variables further to address either the research question in the primary report using alternative analyses or different research questions not considered in the original report. Furthermore, limitations of prior statistical methods can be examined. For example, the inherent defects of stepwise analyses have been exposed (Thompson, 1995, 2006a). When sufficient information is reported, the interested reader can conduct and compare results of stepwise regressions (or other analyses) from prior published reports with more suitable analyses (e.g., all-possible-subsets analyses) to better investigate research phenomena. The inclusion of matrix summaries, means, and sample sizes enables researchers to explore further the nature of constructs (e.g., EFA) and estimate alternative models (e.g., Kline, 2005). The secondary analysis reported by Marsh, Dowson, Pietsch, and Walker (2004) is an excellent example of the benefits of such secondary analyses.
Meta-Analytic Thinking The advent of technology has facilitated the ease with which researchers can conduct more complicated analyses. In the 1960s, 1970s, and 1980s, limitations in software hindered conducting complicated analyses. For example, in the early days of SEM, researchers had to be masters of both matrix algebra and virtually the full Greek alphabet! However today, with means, sample sizes, and matrix summaries, the same analyses—regardless of when the article was published—can be conducted and results compared across studies and time. Unfortunately, reporting correlation matrices has not been the common practice in many journals; therefore, the researcher’s ability to compare studies over time on some topics may be limited.
Future Enforcement of Reporting Standards The present report illustrated the value and benefits that reporting means, sample size, and matrix summaries can bring to secondary research, particularly with regard to encouraging the use of multivariate methods and meta-analytic thinking. Hopefully, our illustrations will also entice readers’ curiosity about the rationale underlying the development of standards and encourage further exploration as to how adhering to the reporting standards can benefit the research community.
Received for publication February 17, 2009. Revision received May 5, 2009. Accepted for publication May 7, 2009.
Educational Researcher, Vol. 38, No. 5,
343-352 (2009)
 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
|
 |
|
Sign In to gain access to subscriptions and/or personal tools.
TOP
Abstract
Benefits to Secondary Research
2)" (
variable. In regression, no multiplicative weight is explicitly applied to the outcome variable Y scores, but regression analysis can be conceptualized as also applying a weight to the outcome variable, albeit always a multiplicative weight of 1.0. 
