Jumping to Conclusions

In a recent post on Education Week's Inside School Research blog, Debra Viadero offers a caution about President Obama's support for community colleges. Pointing to her recent article on community college research that indicated how much more we need to know about how best to improve completion rates in that sector, she questions whether the president would be wiser to place his bets on career colleges. She says that a recent study by the Educational Policy Institute (EPI) and an ongoing program of research by James Rosenbaum and colleagues support her contention that community colleges ought to take cues from career colleges.

In my opinion, this talented reporter is jumping to conclusions.

Yes, the graduation rates at two-year for-profit colleges exceed those at two-year public colleges. No one disputes that. That does not necessarily mean, however, that career colleges are outperforming community colleges, or that community colleges should take steps to become more like career colleges. The plausible alternative explanations for the differences in results are numerous. For example, the students attending the two types of colleges may differ in important yet unmeasured ways, ways that are associated with chances of graduation (what researchers refer to as selection bias). Is one group more economically or educationally advantaged? More motivated? More apt to have a family, nighttime work, or receive tuition support from an employer? It's also possible that the differences in graduation rates stem from constraints that community colleges face but career colleges do not-for example, inadequate resources or a lack of control over mission or governance. It's one thing to point to differences in practices between the community colleges and for-profit colleges, and another thing to attribute those differences to variation in the "will" or intentions of practitioners, rather than attribute them to under-funding and all that comes with it.

Establishing that community colleges have poorer graduation rates than career colleges for reasons they can and should do something about requires evidence that the two differ on one or more key aspects that is causally linked to college completion. Say we knew that smaller classes caused better student retention-and community colleges have larger classes than career colleges. We'd then be able to say, there's something community colleges ought to fix. But we don't have evidence that that's the case.

Instead, research simply establishes that (a) career and community colleges have different graduation rates and (b) career and community colleges (sometimes) employ different institutional practices. Rosenbaum and his colleagues have done a nice job, as Viadero notes, of documenting the latter-using qualitative methods, mostly at colleges in the Chicago area. But they have not shown that those practices cause observable differences in graduation rates. Moreover, while they've produced one paper indicating that differences in the student populations at career and community colleges do not appear to account for disparities in outcomes, that analysis is based solely on a limited set of observable characteristics-and therefore don't rule out the possibility that different levels of student motivation, for example, are really the culprit. Just think about how students get to college-many at career colleges are actively recruited (sometimes off their living room couches) while many at community colleges effectively wander in the door. Why would we think, then, that career and community colleges are serving the same kinds of people and producing different results?

There's another consideration Viadero neglects, and that's college costs. Students at career colleges leave with far more debt than students at community colleges. Data from the 2007-2008 National Postsecondary Student Aid Study reveal that 61% of community college students graduate with less than $10,000 in debt, compared to only 22% of students graduating from 2 year for-profits. In contrast, 19% of graduates from 2 year for-profits have $30,000 or more in student loans, compared to only 5% of community college graduates. Nearly all students (98%) finishing at 2 year for-profit colleges have taken on a loan, compared to just 38% of community college graduates. Is that a problem? Is it offset by higher rates of graduation? The answers are far from clear. Absent better ones we shouldn't be relying on evidence like EPI's---a study of career colleges' high graduation rates that was supported by the Imagine America Foundation, formerly the Career College Foundation, established in 1982 as the research, scholarship and training provider for the nation's career colleges. Full-text of that study wasn't even placed online for researchers to fully vet!

Community colleges have a long, rich history of serving this nation. Sure, there's room for improvement, but without more solid evidence of which changes are needed let's not jump to conclusions and tout the for-profits as a model to which they ought to aspire. We might end up in a bigger mess than we're already in.

POSTSCRIPT:

I have now obtained a copy of the full EPI report. My suspicions were correct: the authors use nothing more than simple descriptive comparisons of students' characteristics and degree completion rates (calculated using NCES's DAS system, likely without propering weighting) to support their causal claims about the "benefits" of attending community college. For example, they write "The report suggests that career colleges work harder to provide appropriate student services and support" but present no data on institutional services or effort expended, particularly any tied to student outcomes. Their final conclusion -- "statistically, not only do students attending career colleges perform as well as or better than many other students attending comparative public institutions, but they persist in and complete their education while typically being more economically, educationally and socially challenged than other students"-- is based on nothing more than comparisons of sample means (no regression, no nothing). C'mon folks, this ain't the kind of research any consumer ought to be taking seriously. Glad to see Kevin Carey agrees.

State Teacher Policies Suck!

I'm sure glad that Kate Walsh and company weren't my professors in college. Damn! They are tough graders! With the exception of eight southern states (Alabama, Arkansas, Florida, Georgia, Louisiana, South Carolina, Tennessee, Texas) that received a 'C' and three northern states (Maine, Montana, Vermont) that received a 'F', every U.S. state received some version of a 'D' in the latest edition of the National Council on Teacher Quality's State Teacher Policy Yearbook. In grading the states, the authors look at five broad teacher quality areas (and numerous metrics within them): teacher preparation, expanding the pool of teachers, identifying effective teachers, retaining effective teachers, and exiting ineffective teachers.

While it is easy to poke holes at some of the National Council on Teacher Quality's seemingly ideologically-driven work (such as, I believe, its excessive focus on teacher pensions), much of its state policy analysis has a strong foothold in research and is one of the most comprehensive and regular analyses of state teacher policies. Like it or not, there is an increasing alignment between the NCTQ's scorecard and that employed by the U.S. Department of Education in the Race to the Top competition. The entire report should not be dismissed because of who they are (or are perceived to be). States should feel challenged by some of the analysis within the Yearbook and should consider looking to the "best practice" states identified under some of the metrics.

Here's a brief summary of the report's findings:

• State teacher policies are "broken, outdated and inflexible."
• Evaluation and tenure policies take too little or no account of classroom effectiveness. 47 states "allow tenure to be awarded virtually automatically."
  
• States are "complicit" on keeping ineffective teachers in classrooms. Only 1 state separates dismissal policy for poor performance from criminal and moral violations.
• Few states provide robust enough alternate routes into teaching.
• States' requirements for elementary teacher, middle-school teacher and special education teacher preparation are inadequate.
• There is too little accountability for teacher preparation in state policy. Only 5 states set minimum standards for teacher preparation program performance.
• States "cling to outmoded compensation structures," including the single salary schedule.
  

My primary quibble with the report is that it appears to completely and utterly discount the role of induction, mentoring and professional development in strengthening teacher effectiveness. Even if we prepare teachers better, recruit non-traditional candidates into the profession, retain them longer, compensate them differently, make evaluations more regular and meaningful, and find appropriate ways to terminate the small fraction of truly incompetent ones, it still will not be enough to maximize teacher effectiveness. There will continue to be a need for high-quality, individualized support upon entry into the profession and regular opportunities for data-driven, instructionally-focused professional development through a teacher's career. Professional development is not featured as a metric in the report at all and induction only enters as a criteria with regard to teacher retention, rather than teacher effectiveness -- which is where its most important power truly lies. That said, the evaluative criteria the report lays out about induction policy (on page 183-184 of the printed report) are worth noting and includes elements that states must attend to: mentoring of sufficient frequency and duration, mentoring provided at the start of the school year, and attentive mentor selection and high-quality training.

I won't beat this horse any further today, but check out these past posts for greater substance on what I'm getting at here with regard to the inadequate focus on the developmental needs of new and veteran teachers:

Race To The Top: Under The Hood
RttT: Redefining Teacher Effectiveness
Measurement Is Not Destiny

In other news, experts are doubting the likelihood of a 2010 reauthorization of the Elementary and Secondary Education Act, so these state teacher policies with an added dose of Race to the Top reforms is likely to be where it's at over the next year plus.

Making SAFRA Count

The end of last year was a busy time for me as I waited out the birth of my daughter who decided to spend an extra 10 days lounging in utero before emerging into the Wisconsin winter. I was so focused on strategies to promote her exit (sidenote: talk about an area in need of better research-give gobs of data on live births for hundreds of years, docs still refuse to hazard a prediction of labor occurring on any given night!), I virtually shut out the world of higher education policy. Imagine!

Thankfully, others were hard at work around and over the holidays, thinking about ways to make sure that the substantial, timely, and hard-won investment which will (fingers crossed) soon come to higher education via the Student Aid and Fiscal Responsibility Act (SAFRA) are most effective. Evidence of that work is contained in a December Lumina Foundation memorandum to the U.S. Department of Education, awkwardly (but accurately) titled "Structuring the Distribution of New Federal Higher Education Program Funding to Assure Maximum Effectiveness."

The memo gets it (mostly) right. There's great potential for this money to count, but also a real possibility it will do next to nothing if mismanaged. For example, if definitions of key terms like "college completion" are vague, and standards for "rigorous" research evidence ambiguous, then funds will likely go to continuing business as usual-for example, supporting programs that purport to increase college access while doing little to change rates of success-leading some to ask, access to what?

To avoid this the Department of Education needs a distribution system based first and foremost on one principle: keep it simple. It should make states define college completion and disseminate that definition-then stick to it. It's easiest to tell if plans are straightforward and consistent with intended principles if prospective grantees are forced to explain their ideas in a concise manner. Lumina gets this, and its team recommends a two-step process that requires a concept paper in advance of a full proposal.

So the good news is that this Lumina paper hits many of the key issues and makes some solid recommendations. That said, its authors missed an opportunity to address one important issue. The section titled, "How will the U.S. Department of Education know if these investments are actually helping to meet the President's goal?" is essential. It goes to the heart of one major goal of SAFRA-to increase the body of knowledge about what works in promoting college completion, and therefore the field's capacity to create lasting change.

As I recommended to ED's Bob Shireman early last year, we can do higher education a great service by holding a high bar for what constitutes research on college completion. Too often research in higher education hypothesizes that policies or practices advance desired outcomes, but utilizes insufficient methods to establish causal linkages between the two. As a result, we often don't know whether the results we see can be directly attributed to the new practice or investment.

In this case, ED should define "research" and "researchers" and "evidence," ideally in ways that are consistent with current practices at the Institute for Education Sciences; and require states to use those definitions. There should be a prescriptive process for selecting researchers (so as to make sure that truly independent evaluations are conducted) and proposals that allow for sustained research should be prioritized (e.g. those that leverage supportive foundation funding to continue the work to assess mid and long-range outcomes). I'd also like to see ED involved in increasing the capacity of researchers to do this kind of work, since it's far from clear how the demand for new work can be met by the current supply of higher education researchers. Maybe an IES pre- and/or post-doc training program targeted to postsecondary education?

Sure, this would require setting aside sufficient funds for the research side of the initiatives-but absent that investment, we'll likely never know whether the money spent on SAFRA-funded programs and policies had any real effect. That would, of course, be business as usual-precisely what we must avoid if we want to make this once-in-a-lifetime opportunity really count.