Means and Ends: A Comparative Study of Empirical Methods for Investigating Governance and Performance

Carolyn J. Heinrich and Laurence E. Lynn, Jr.

Background

Empirical studies designed to analyze governance relationships in the public sector generally focus on program processes or outcomes at a single organizational level. While some studies group individuals and attempt to explain average affects or outcomes, other studies analyze the influence of organizational or structural factors on individual outcomes. This Harris School paper titled, Means and Ends: A Comparative Study of Empirical Methods for Investigating Governance and Performance, by University of Chicago researchers Carolyn J. Heinrich and Laurence E. Lynn, Jr., focuses on the use of multilevel models to formulate and examine hypotheses that investigate the interaction between factors measured at one level of an administrative hierarchy and variables at another level.

Recent advances in statistical methodologies enable researchers to conduct empirical analysis of factors interacting at multiple levels of hierarchy within government and social systems. Heinrich and Lynn examine the application of multilevel models and alternative modeling strategies in three fields: education, drug abuse treatment, and employment and training programs.

Focusing on the administrative structures and management incentive policies of Job Training Performance Act (JTPA) programs, Heinrich and Lynn compared multilevel modeling techniques with other frequently used approaches. Data collected during the National JTPA Study on individuals' characteristics, earning and employment outcomes were combined with administrative and policy data that were also obtained from the 16 study sites over a three-year period. Heinrich and Lynn also drew data from a local study of JTPA training service provider contracts to employ in this research. Results from both studies indicated that site-level administrative structures and local management strategies (including performance incentives) had a significant influence on client outcomes.

Methodology and Comparison of Statistical Models

The data used in these two JTPA studies allowed for a comparison of different statistical approaches. The three statistical approaches examined were: hierarchical linear models (HLM), ordinary least squares (OLS) regression models using individual-level data, and OLS models using outcome measures aggregated at the site or administrator level. For OLS regressions of individual-level outcomes, the site-level administrative and policy data were linked to the records of the individual participants. This meant that all participants in a given site and year had the same site-level variable values.

Research Summaries are designed to help broaden the dissemination of policy-related research. Research Summaries are funded by the Irving B. Harris Graduate School of Public Policy Studies at the University of Chicago.

For the site-level OLS regressions, the individual-level data were collated by site or by contract, and average measures of these variables were entered into the models, coupled with the site-level administrative and policy variables. In the hierarchical linear models (HLM), each of these two levels of data was formally represented by its own sub-model, with each sub-model specifying the structural relations occurring and the residual variability observed at that level.

The presence of significant intra-level correlation in hierarchical data violates basic assumptions of the OLS regression model. Among these is the independence of observations and that the number of independent observations is equal for all variables. Hierarchical linear models provide for partitioning variance into components associated with the different levels of analysis. Subsequently, they allow the detection and exploration of differences across contexts or groups.

Key Findings, Job Training Performance Act (JTPA) Programs

Upon comparing the modeling strategies, Heinrich and Lynn found that in some models, the HLM and the individual-level OLS estimated variable coefficients are very close for both individual-level and site-level predictors. This confirms that where a very small percentage of variation occurs at the site-level (approx. 3%) OLS and HLM methods are likely to produce comparable estimates of individual- and site-level effects. However, using HLM in these cases enables researchers to identify how much of the variation in outcomes lies at the different level of analyses. It also allows researchers to assess what proportion of this variation is explained by the models specified, and whether any statistically significant variation remains to be explained further.

Table 1: Empirical estimates of governance variables* in hierarchical linear and OLS models of JTPA participants' first post-program year earnings outcomes (using National JTPA Study data)

Predictors - (site level)	Hierarchical linear model	Individual-level OLS	Site-level OLS (average)
PIC is the administrative entity	1737.40*** (4.59)	1727.15*** (4.74)	745.12*** (2.98)
PIC and LEO/CEO are equal partners	-1933.65*** (-2.61)	-1949.44*** (-2.73)	-255.88 (-0.56)
Percent of services provided directly by administrative entity	-2618.57** (-2.26)	-2604.93** (-2.35)	-564.00 (-0.43)
Percent of performance-based contracts	-2719.45*** (-2.60)	-2709.02*** (-2.69)	-2033.00* (-1.80)
Weight accorded to employment rate standard	15887.75*** (3.93)	15888.00*** (4.09)	15710.00*** (3.13)
Minimum number of standards sites must meet to qualify for performance bonuses	22.25 (0.39)	21.50 (0.40)	11.74 (1.16)
Requirement that performance bonuses must be used to serve highly disadvantaged groups	-866.66** (-2.30)	-865.32** (-2.36)	-258.00 (-1.35)
Model predicting power - percent of variation explained by model	13% individual-level; 97% between-site	adjusted R2 =13.2%	Adjusted R2 = 87.6%

* Only governance variables included in the model are shown. The full set of model results may be requested from the authors.

When there are stronger cross-level effects present in the data, as in the study of JTPA service provider contract administration, even HLM and the individual-level OLS variable coefficient estimates diverge.

Table 2: Empirical estimates of governance variables* in hierarchical linear and OLS models

of JTPA participants' hourly wages at termination (study of service provider contracts)

Predictors - (contract level)	Hierarchical linear model	Individual-level OLS	Site-level OLS (average)
Private, nonprofit contractor	0.29* (1.84)	0.10* (1.84)	0.15* (1.84)
For-profit contractor	0.77*** (4.10)	0.40*** (4.10)	0.11*** (4.10)
Performance incentives in contract	0.34*** (2.72)	0.32*** (2.72)	0.09*** (2.72)
Random effect: Participants under age 18 years served and performance incentives in contract	0.30*** (2.67)	n.a.	n.a.
Model predicting power - percent of variation explained by model	9% individual-level; 68% between-site	Adjusted R2 = 34.7%	Adjusted R2 = 68.2%

39.1% of total variation occurs between contracts; 18.5% (statistically significant with p=0.001) remains to be explained in conditional model.

* Only governance variables included in the model are shown. The full set of model results may be requested from the authors.

In contrast to the more comparable findings of the HLM and individual-level OLS models, site-level models produced inconsistent results. The sizes and signs of some individual- and site-level coefficients (not all are shown in the tables included in this summary) were contrary to both theoretical expectations and the findings of previous empirical studies. These results imply that modeling administrative processes and program outcomes across multiple sites, using data on clients aggregated at the site level, is a less reliable approach than similar (multi-site) client-level data analysis. Krull and MacKinnon (1999), who also compared multilevel modeling strategies to individual- and group-level OLS regressions, emphasize that when individual-level data are aggregated, researchers should expect that individual and group level analyses of the same data might indicate relationships that differ in both magnitude and direction, since the ability to predict individual-level variation in these models is eliminated. They similarly concluded that "multilevel-based estimates of the standard error showed considerably less bias than OLS-based estimates," and that OLS analyses were less efficient than multilevel analyses (433).

The inconsistencies in the site-level policy/administrative coefficients are of particular importance for the study of governance, as these variables are nearly always the primary focus of public policy studies. Many of these studies use site-level approaches, and commonly report high levels of variation, explained with a relatively small number of policy related variables. The findings presented in this paper emphasize that ignoring the variation in individual-level outcomes and the potential cross-level effects between variables operating at individual- and site-levels may lead to inaccurate estimates of policy related variables' effects.

While creating and/or re-analyzing data sets might not always be feasible, the results of this study suggest some advantages of the HLM approach. Multi-level modeling strategies are more likely to produce unbiased estimates of the influences that policy, administrative and/or structural variables have on program participants, compared to traditional ordinary least squares regression models. In addition, the HLM approach may produce a broader comprehension of complex, hierarchical relationships. It also yields more information about the amount of variation explained by statistical models at different levels of analysis and increases the ability to generalize findings across different sites and organizations. Heinrich and Lynn also stress, however, that modeling approaches should not be selected solely to 'fit' the data but must also appropriately address the question at hand.

 

Research Summaries are designed to help broaden the dissemination of current policy-relevant research. These Summaries are funded by the Irving B. Harris Graduate School of Public Policy Studies at the University of Chicago.

For more information, contact Jamie Rosman at HarrisSchool@uchicago.edu or (773) 702.2287.