Strengthening Student Support: A Sensible Proposal with What Results?

Cross-posted from Brainstorm

Anyone who's taken a hard look at the reasons why more students drop out of community college realizes it's got to have at least something to do with their need for more frequent, higher-quality advising. After all, in many cases these are students who are juggling multiple responsibilities, only one of which is attending college, and they need to figure out a lot of details-- how to take the right courses to fit their particular program (especially if they hope to later transfer credits), how to get the best financial aid package, how to work out a daily schedule that can maximize their learning, etc. It's fairly easy to figure that in fact community college students would likely stand to benefit more from good advising than their counterparts at many 4-year institutions.

Except high-quality advising isn't what they get. Counselor-student ratios are on average 1000:1. That's right-- one counselor for a population the size of a decent high school. In elementary and secondary schools the ratio is 479:1. There's a pay disparity as well-- in k-12 the Bureau of Labor Statistics reports that the median annual earnings for a counselor in 2006 was nearly $54,000. For counselors at community colleges it was $48,000 (and for those at other colleges it was $42,000). Now, perhaps the salary differentials reflect the different work load, and assumptions about it being easier to counsel adults. But I tend to think this is offbase-- these are outdated notions of who community college students are and what they need.

So what would happen if we reduced the counselor/student ratio at community colleges to a standard even better than the national average in k-12? And at the same time ramped up the intensity of the counseling? Theory would suggest we should see some meaningful results. Many studies, including my own, point toward a persistent relationship between parental education and college outcomes that's indicative of the importance of information-- and information (plus motivation) is what counseling provides. So, putting more counselors into a community college and increasing the quality of what they provide should work-- if students actually go and see them.

To test these hypotheses, MDRC (a terrific NYC-based evaluation firm) recently conducted a randomized program evaluation in two Ohio community colleges. In a nutshell, at college A students in the treatment group were assigned (at random) to receive services from a full-time counselor serving only 81 students, while at college B students in the treatment group had a counselor serving 157 students. In both cases, the control group students saw counselors serving more than 1,000 students each. In addition to serving far fewer students than is typical, these counselors were instructed to provide services that were "more intensive, comprehensive, and personalized." In practice, students in the treatment group did see their counselors more often. The "treatment" lasted two semesters.

The students in this study are Midwesterners, predominantly (75%) women, predominantly white (54%), with an average age of 24, half living below the poverty line and half are working while in school. I think it's also worth pointing out that while all applied for financial aid, these were not folks who were overwhelming facing circumstances of deprivation-- 88% had access to a working car, and 64% had a working computer in their home. And 98% were U.S. citizens.

The results indicate only modest results. After one semester of program implementation, the biggest effects occured-- students in the treatment group were 7 percentage points more likely to register for another semester (65 vs. 58%). But those differences quickly disappeared, and no notable differences in outcomes like the number of credits taken and other academic outcomes occured. Moreover, the researchers didn't find other kinds of effects you might expect--such as changes in students' educational goals, feelings of connection to the college, or measured ability to cope with struggles.

So what's going on? The folks at MDRC suggest 3 possibilities: (1) the program didn't last long enough to generate impacts, (2) the services weren't comprehensive enough, (3) advising may need to be linked to other supports--including more substantial financial aid--in order to generate effects. I think these are reasonable hypotheses, but I'd like to add some more to this list.

First and foremost, there's a selection problem. MDRC tested an effect of enhanced advising on a population of students already more likely to seek advice-- those who signed up for a study and more services. Now, of course this is a common problem in research and it doesn't compromise the internal validity of the results (e.g. I'm not saying that they mis-estimated the size of the effect). And, MDRC did better than usual in using a list of qualified students (all of whom, by the way had to have completed a FAFSA) and actively recruiting them into the study-- rather than simply selecting participants from folks who showed up to a sign-up table and agreed to enter a study. But, in the end they are testing the effects of advising on a group that was responsive to the study intake efforts of college staff. And we're not provided with any data on how that group differed from the group who weren't responsive to those efforts--not even on the measures included on the FAFSA (which it seems the researchers have access to). Assuming participants are different from non-participants (and they almost always are), I'm betting the participants have characteristics that make them more likely to seek help-- and therefore are perhaps less likely to accrue the biggest benefits from enhanced advising. I wish we had survey measures to test this hypotheses-- for example we could look at the expectations of participants at baseline and compare them to those of more typical students-- but the first survey wasn't administered until a full year after the treatment began. To sum, up, this issue doesn't compromise the internal validity of the results, but it may help explain why such small effects were observed-- there are often heterogeneous effects of programs, and those students for whom you might anticipate the bigger effects weren't in the study at all.

A second issue: we just don't know nearly enough about the counterfactual in this case-- specifically, what services students in the control group received. (We know a bit more about differences in what they were offered, e.g. from Table 3.3, but not in terms of what they received,) We are provided comparisons in services received by treatment status only for one measure-- services received 3+ times during the first year of the study (Appendix Table c.3), but not for the full range of services such as those shown in Appendix Table C.1. For example we don't know that students in the control and treatment groups didn't have similar chances of contacting a counselor 1 or 2 times, only the incidence of 3+ contacts. If the bar was rather high, it may have been tougher to clear (e.g. the treatment would've needed to have a bigger impact to be significant).

Having raised those issues, I want to note that these are fairly common problems in evaluation research (not knowing much about either study non-participants or about services received by the control group), and they don't affect MDRC's interpretations of findings. But these problems may help us understand a little bit more about why more substantial effects weren't observed.

Before wrapping up, I want to give MDRC credit for paying attention to more than simply academic outcomes in this study-- they tested for social and health effects as well, including effects on stress (but didn't find any). As I've written here before, we need to bring the study of student health and stress into educational research in a more systematic way, and I'm very glad to see MDRC doing that.

So, in the end, what have we learned? I have no doubt that the costs of changing these advising ratios are substantial, and the impacts in this case were clearly low. Right now, that doesn't lend too much credence to increasing spending on student services. But, this doesn't mean that more targeted advising might not be more effective. Perhaps it can really help men of color (who are largely absent from this study). Clearly, (drumroll/eye-rolling please), more research is needed.

RttT: Redefining Teacher Effectiveness

My colleagues and I at the New Teacher Center have offered up what I believe to be a balanced and thoughtful series of recommendations to strengthen the teacher and principal effectiveness provisions in the U.S. Department of Education's proposed Race to the Top regulations. You can find the NTC's initial public comments -- submitted on August 21 -- here. And you find an addendum -- filed yesterday -- offering recommendations for specific language additions, here.

Generally, we are supportive of the overall direction of Race to the Top. But we feel that its focus on teacher effectiveness is too narrowly about measuring individual teacher impact at the exclusion of supporting all educators to strengthen their teaching and leadership skills and attending to teaching and learning conditions within schools that impact student success.

Here is a brief summary of our recommendations:

Improving Teacher Effectiveness and Achieving Equity in Teacher Distribution
• The RttT guidelines should include a definition of teacher effectiveness that acknowledges and
supports the development of teacher and principal practice, especially during the early years.
New teachers and principals, who disproportionately work in struggling schools, need strong
mentoring and support to become effective.

• The RttT guidelines should define ‘effective principal’ more expansively, drawing upon
additional measures of student success and data on teaching and learning conditions to fully
reflect the impact of teachers, school leaders, and school environment on student learning.

• The RttT guidelines should require states to address school leadership development and teaching and learning conditions in their strategies to improve teacher effectiveness and the equitable distribution of quality teachers.

Improving Collection and Use of Data
• RttT guidelines should specifically include teaching and learning conditions data gathered from
practitioners to help schools, districts and states better understand supports and barriers to
teacher effectiveness and equitable teacher distribution, and to incorporate this information into
their longitudinal P-20 data systems.

And here is some selected language that provides insight into our thinking around teacher effectiveness and teacher development:

Teacher effectiveness in the proposed RttT guidelines focuses exclusively on value-added student assessments. While value-added student achievement data can be used to reward and recognize certain achievements by educators, it should not be the sole method by which teachers are evaluated, observed, rewarded, and deemed “effective.” Firing the least effective teachers and rewarding the most effective alone is short-sighted and ignores the vast majority of teachers in the middle who can achieve greater success if given access to high-quality induction and professional development, strong and supportive school administrators, and opportunities for collaboration and leadership. Great teachers are made – not born. Teachers need professional support and opportunities to develop their practice, including focused induction during their initial years in the profession. It is important to measure teacher impact on student learning, but measuring impact without providing the means to help educators strengthen their practice will ultimately fail our schools.

If RttT is to be an effective reform strategy, it needs to recognize teacher development as a primary means to maximize classroom effectiveness. RttT should require states not merely to identify the best teachers, but see that their successes form the building blocks of a better understanding of effective teaching practice that can be replicated in classrooms across America.

And on teaching and learning conditions:

In order for school leaders to attract and retain quality teachers, research shows the need for school leaders to make decisions based on data that incorporate the perspective of classroom teachers. Teacher survey data can provide insight into the school culture, how decisions are made, and the use of instructional and planning time for teachers. Such contextual data may explain differences in teacher effectiveness between schools and districts. NTC has worked with over 300,000 educators in 10 states, and collected teaching and learning conditions data from over 8,000 schools to utilize in school improvement plans. In North Carolina, the State Board of Education now requires schools to utilize the data from the biennial working conditions survey to inform annual improvement plans and strategies.

Quality teachers will seek out and stay with strong supportive school leaders; therefore, using RttT funds for salary bonuses in hard-to-staff schools would not be the most effective approach. RttT should encourage states to show how they are using data from teachers, along with student achievement and other relevant data, to develop policies for these schools, strengthen school leadership, and ensure that they are settings where the most effective teachers want to work and can succeed.

The RttT public comment period closes today and a spate of organizations have submitted comments just under the wire. They range from narrow to broad, supportive to critical, and offer everything from research-based suggested line edits to what basically look like press releases buttering up Secretary Duncan.

Visit here to review all of the public comments submitted.

Is it Time to Get Onboard with Online Education?

In several recent interviews and blog posts I've expressed my hesitation about the move toward online learning in higher education. My concerns are fairly common ones and go like this: How do we know that students are engaged, or even awake, when participating online? How do we know that online learning is as effective as classroom learning? How do we know that any negative consequences outweigh the cost savings? And what exactly are those cost savings? (After all, technology isn't cheap) And finally, despite claims to the contrary, the digital divide still exists-- so how do we know that low-income and rural populations will get the access to online learning they need?

Admittedly, I'll always be forced to note that for most of these big questions there's little evidence to the contrary-- e.g. we don't know much about the effectiveness of classroom learning in higher education either, we don't know its relative cost-effectiveness, and we don't know how many are left out of higher education because they can't make it to a classroom setting.

But, in this case I've tended toward the traditional and in some sense the sociological-- prioritizing the value of in-person face-to-face social interactions over online ones, and assuming that more mentoring occurs in an in-person relationship, adding value to the instruction. So, I tend to say things like "the move to online education is premature" and "we need more evidence."

Ok, so this summer the U.S. Department of Education came out with a decent response in the report "Evaluation of Evidence-based Practices in Online Learning: A Meta-Analysis and Review of Online Studies." It came out in May-- yes, I'm late to the game here (but honestly, the thing is 93 pages and I read it cover to cover before writing this post). In typical What Works Clearinghouse fashion, the authors pay detailed attention to the methods used in each study they reviewed, and I'm very comfortable with the standards of evidence employed (though I have to note, not every study was peer-reviewed--many were dissertations). They also took care to distinguish between the populations considered in each study (e.g. k-12 versus higher education), and the type and quality of online education examined.


This report taught me the following: (1) There's been much more assessment of the effectiveness of online learning in higher education, compared to k-12. (2) Student outcomes of online learning seem to be somewhat better than those of classroom learning-- but it's not exactly clear that apples and apples comparisons are being made, mainly because the amount of actual instructional time in online courses is greater than that in classroom settings. As the authors write, "Despite what appears to be strong support for online learning applications, the studies in this meta-analysis do not demonstrate that online learning is superior as a medium, In many of the studies showing an advantage for online learning, the online and classroom conditions differed in terms of time spent, curriculum and pedagogy. It was the combination of elements in the treatment conditions (which was likely to have included additional learning time and materials as well as additional opportunities for collaboration) that produced the observed learning advantages."

In some sense this is the kind of evidence I wanted to see in order to be a bit more comfortable with the accelerated pace towards online education. Yet at the same time, I'm not convinced. Apart from the caveat regarding the actual medium, mentioned by the authors above, another reason is that while the authors are right that most of the studies of online learning aren't in k-12, it's also clear from reading the bibliography that they're not in typical undergraduate education either. The meta-analysis is dominated by studies of students in undergraduate education, yes, but of students who have declared their major in undergrad and are taking a specific kind of course (e.g. nursing). (I count only 5-7 studies in this meta- analysis that involve more entry-level courses, or those for struggling learners.) I'd argue this is a very specific, more highly motivated group of adult learners than the folks that a scale-up of online instruction in undergrad education is bound to reach.

Reading between the lines a bit, it seems clear that U.S. DOE won't be motivated to fund more evaluations of online learning outside of k-12 in the near future. I think that would be a mistake- we need to know more about which kinds of online learning work for which undergraduates and under what conditions. We also need to know more specifics about both costs and impacts, allowing for judgements to be made in a cost-effectiveness framework. In the meantime, however, I'm a bit more convinced that online ed is a reasonable way to move forward in solving crowding problems in specific majors, particularly with more advanced students.