Assessment: Beyond the term paper

From Campus Technology:

In designing courses for online environments we have been somewhat successful at scuttling closed-book exams for assessing student learning--largely due to the challenges of monitoring exams. However, we still appear to be stuck in another very entrenched mode of assessing student learning: research papers and project reports.

Why is the traditional paper so prevalent in assessment, and how can we move beyond it to alternative evidence of student learning? And how can we leverage technology and new media in our assessment strategies?

And now, for the rest of the story...

Why I like assessment...

Some quotes from an article in the Chronicle of Higher Education:

I like it because it encourages faculty members to think more carefully about what they do, how they do it, and why they do it that way. I like it because it helps raise questions about how our teaching strategies affect learning outcomes. And I like it because in the process, we discover more about how our teaching fits in with programs and curricula beyond our own courses. Good-quality assessment simply asks about our goals, our instructional procedures, and the link between both of those and learning.

Assessment can help. It can teach faculty members to work together to teach and assess those learning goals. For example, many sociology programs stress the role of research methods across courses, but my interviews with students suggest that students generally fail to apply their knowledge of those methods in other courses. In part that happens because instructors do not reinforce such knowledge and skills. Assessing both the courses and students' knowledge will highlight such gaps and help transform their cumulative experience by encouraging instructors to improve both individual courses and the learning gained across courses.

The entire department would benefit as all courses became part of a well-thought-out whole. Professors gain classes full of prepared students, and students report their highest levels of satisfaction and learning in departments where faculty members collectively assume responsibility for the entire curriculum and its assessment. It takes a village of engaged faculty to raise successful students. That same village can provide better assessment than can one designated person, and can make better use of the results.

Let's not do assessment just because it is mandated. Let's not do it to make accreditation agencies happy or because everyone else is doing it. Let's do it to improve learning.

And now, for the rest of the story...

Closing the gap between teaching and assessing

More from the editors of the new book mentioned below:

Good assessment can give us concrete information about whether students are learning, how much they are learning, and in what areas. And there are institutions, faculty members, and administrators that not only know this body of work but make good use of it as they seek to strengthen their institutions and serve their students better. All too often, though, there is a considerable gap between institutional assessment and teaching. Some faculty members embrace assessment efforts, some are highly critical of them, but most, perhaps, are barely aware of them.

Some things, faculty quickly discover, are easily measured: In the field of literary study, we can gauge students' skill as grammarians, even their ability to construct a persuasive argument. But can we get at the kind of learning that matters most? The kind of learning that leads to full engagement with a topic and a nuanced understanding of its meaning? We will not be able to achieve this until we think about the particular forms of engagement that draw us to deep learning within a discipline, and to the connections between those disciplines and larger social contexts as well as institutional goals.

Exploring forms of alignment among disciplinary goals, institutional aims, and broader social contexts will also better engage faculty in the assessment process. If faculty members were talking regularly with assessment researchers and practitioners, they would have a voice in emerging national conversations about how students learn in different disciplines and what strategies bring student learning to the highest possible levels. They would have a voice in saying what kind of learning really matters in their fields—what outcomes need to be measured—and a chance of aiding in the development of assessment methods genuinely suited to what they teach. They would, in other words, be a guiding force in the work that is a necessary first step in improving learning.

In our research, we insist that every point find solid evidentiary support. Most of us teach our students to do the same: "Can you back that up?" "On what basis do you reach that conclusion?" Yet when it comes to whether or not our students are learning, we rely on evidence that is dubious (teaching evaluations) or circular (grades). Or we abandon the Enlightenment altogether and lapse into faith: We just know. Again, while gut feeling is a crucial part of inquiry, most of us have been rigorously trained to interrogate both received wisdom and unexamined assumptions in our scholarship. Why should our approach to student learning be different?

The clearer we are about our goals for learning, and the better we are at seeing whether we are meeting those goals, and then proceeding—on the basis of that evidence—to strengthen teaching and learning in our classrooms, the better our students will do. With improved learning, we also ensure the viability of higher education and of specific areas of study. Solid data on what students are learning demonstrate the value of a field.

...higher education in general, and the liberal arts in particular, are now under attack in ways that we do not need to explain for most readers of The Chronicle. In that context, resisting efforts to figure out how well our students are learning for the purposes of improvement seems counterproductive. Many academic professional organizations are wisely encouraging their members to reach out to the public and explain the value of their pursuits. We want to remind our colleagues, however, that you don't have to be on the Today show or NPR to be talking to the public. Faculty are already doing this every day, engaging groups of people who will have a disproportionate influence in society compared with their peers who are not going to college. We need to be thinking collaboratively about how best to educate them.

And now, for the rest of the story...

Assessing the ineffable

From Inside Higher Ed, an interview with editors of a new book from the Teagle Foundation on assessment of the hard-to-assess in the disciplines. Remarkably, this book is available for free download on the Teagle web site.

Humanists are sometimes wary of assessment because they feel that there is something ineffable at the heart of their work in the classroom. We realized that the ineffable itself -- in the form of theories of the sublime -- is a subject that we think about a lot. And we wondered whether we could invite people to talk about ineffability -- about the sublime -- as a way of coming to understand some of our most cherished learning goals for our students, and even assessing them. In a way, we were inviting people to face head-on the thing that no one thinks we can talk about. And the collection developed from there.

And now, for the rest of the story...

Trust us; we're experts.

This time I'm just going to post the whole damn thing. It's long for this forum, but compared to other stuff you read, it's short. I started pulling out extracts and realized I was going to end up quoting most of it anyway.

Most of my colleagues are faculty members and that's how I identify myself. I love my profession and I love my academic sisters and brothers. But Geez, people!

Here's today's column in Inside Higher Ed from Dean Dad:

Historiann has a fascinating, and I think largely representative, take on a provocative article in the Washington Post about “fixing” higher education. The original piece outlines eight steps that it argues would make meaningful differences for colleges and universities in the US. Some of them are easy and obvious, like toning down the focus on athletics; others are deeply problematic, like junking merit scholarships. (For my money, there’s something fundamentally wrong when having a good jump shot is a surer ticket to tuition than building a strong record at chemistry or writing.)

The first one is somewhere in between. It’s “measure student learning.” Historiann dismisses this one out of hand, with a quick reference to No Child Left Behind and the following: “Let’s just strangle this one in its crib unless and until we get some evidence that more testing = more education.”

It’s a fascinating response, because it encapsulates so cleanly the unthought impulse that many of us have. Testing equals Republicans equals bullshit; now shut the hell up and write us large checks. Trust us, we’re experts.

It’s written a little more carefully than that, of course, but written specifically to defeat verification. It rejects any sort of “measurement,” but does so by calling for “evidence” that measurement works.

What would that evidence look like? Might it involve, say, measurement? If not, then on what basis could you use a term like “more”? Every meaning of “more” that I can fathom involves some sort of comparative measurement. But to do that, we’d have to agree on a measure. Unless, of course, that was simply a rhetorical flourish, a semi-ironic acknowledgement that such a thing could never be proven because, well, it just couldn’t.

The knee-jerk response to any sort of accountability rests on a tautology. We know better than anyone else because we’re experts; we’re experts because we know better than anyone else. Screw measurement, accountability, or assessment; we already know we’re the best. Just ask us! Now, about that check...

If the folks who care about higher education are even halfway serious about avoiding the traps K-12 is in, the first step is not repeating the same mistakes. “Trust us, we’re experts” simply is not a persuasive argument to the larger public. It may once have been, but it isn’t now, and it hasn’t been for a long time. The difference between Historiann’s perspective and my own is that she seems to assume that failure to defer to rank is the public’s shortcoming; I think it’s basically healthy.

Part of the reason that Academically Adrift has resonated as much as it has, I suspect, is that it argues something that most of us (and most of the taxpaying public) secretly know to be true: many college students skate through without getting appreciably smarter. I consider that a major problem, and one that would require some pretty fundamental structural changes to higher education to address.

Oddly, many of the same people who share Historiann’s dismissal of testing are among the first to decry poor student performance. We expert educators are expert educators, if we don’t mind saying so; therefore, any student failings must...wait for it...be the fault of the students! In fact, they’re getting worse all the time! Now, let’s talk about next year’s tuition increase...

After a few decades of that, the public is getting a bit, well, testy. And well it should.

At base, the popular perception that college is a scam can’t be ameliorated by assertions of expertise, truth, and virtue. If those worked, they would have worked by now. It will be ameliorated, or not, by showing the public some kind of real results. What those results should be is certainly open for debate; as a kid, I remember seeing the space program justified by the development of calculators and digital watches. It might take the form of some sort of exam, or it might take the form of success stories, or it might take the form of new graduates developing wonderful things. Which path to pursue strikes me as a fair and valid discussion. But if we don’t recognize that the basic impulse behind the testiness is essentially valid, we won’t get anywhere. Aristocratic pretensions aren’t gonna cut it; the “appeal to authority” isn’t terribly appealing. We need to show, rather than tell, the public that we’re worth supporting. Which means we need to show ourselves first. Strangling that impulse in the crib is not a serious answer.

"I'm looking thru you; where did you go?"

A "Quick Take" in today's Inside Higher Ed pointed to NILOA's Transparency Framework. At first glance it's not entirely clear what you're looking at. But it seems to be a rather comprehensive guide for presenting student learning outcomes info on an institution's web page. There's a lot of stuff here, and I haven't carefully studied all of it. But it definitely appears to be worth spending some time.

Will Texas take to tuning?

From the Texas Higher Education Coordinating Board's newsletter:

Tuning Oversight 2011 Council for Engineering and Science will apply the “Tuning” and course-level alignment processes to additional engineering and science disciplines

On February 25, with grant support from Lumina Foundation for Education, the Coordinating Board (CB) will host the first meeting of the 2011 Tuning Oversight Council for Engineering and Science, the voluntary faculty advisory council that will assist the CB in continuing to integrate the “Tuning” process (as described below) into the course-level alignment work that was piloted in 2009 through the efforts of the Voluntary Mechanical Engineering Transfer Compact Committee, and which continued in 2010 with the efforts of the 2010 Tuning Oversight Council for Engineering. The 2011 Tuning Oversight Council will consist of four voluntary faculty advisory committees: (1) Biomedical Engineering, (2) Chemical Engineering, (3) Biology, and (4) Chemistry.

Nominations for faculty members to serve on the 2011 Tuning Oversight Council for Engineering and Science have been requested from universities and community colleges across the state; to date, invitations for specific nominees to participate on the council have been sent to 36 faculty members representing 31 institutions. The final makeup of the council and respective committees will be determined by February 18.

The 2010 Tuning Oversight Council for Engineering, which is made up of higher education faculty representing 30 higher education institutions from across the state, is expected to complete Tuning and lower-division course-level alignment work for the disciplines of Civil, Electrical, Industrial, and Mechanical Engineering in May. Additional information is available below and online at: www.thecb.state.tx.us/tuningtexas.

“Tuning” is a faculty-led pilot project designed to provide an indication of the knowledge, skills, and abilities students should achieve prior to graduation at different degree levels.

Tuning involves:

• Faculty from different sectors and institutions agreeing on what students in a field must know, understand, and be able to do;
  
• Surveying faculty to prioritize subject-area competencies;
  
• Soliciting views of students, graduates, and employers on the most valued general competencies in the field;
  
• Faculty defining degrees using active learning outcomes that can be assessed through coursework and other means; and
  
• Mapping the employability of degree holders.

“Fine-Tuning” or alignment of lower division courses involves:

• Identifying common and atypical lower-division courses in a discipline where learning outcomes align with the Tuned discipline;
  
• Reviewing syllabi for all lower-division courses; and
  
• Selecting the most comprehensive course description, pre-/co-requisites, and learning outcomes for each lower-division course.

Speeding without looking

The high speed proliferation of online learning sometimes seems like we're driving 100 mph...with our eyes closed.

Virtual education is in a period of rapid growth, as school districts, for-profit providers, and nonprofit start-ups all move into the online learning world... But without rigorous oversight, a thousand flowers blooming will also yield a lot of weeds.

At present, virtual education lacks a firm understanding of what high performance looks like.

Virtual public education can be delivered by all types of providers, including charter schools, for-profit companies, universities, state entities, and school districts... Such diversity brings challenges.

[V]irtual education lacks a commonly accepted set of quality outcome measures. Quality can't be defined by the design of a school or by inputs alone; instead, it must focus primarily on outcomes.

[This] complexity can't be an excuse for inaction. Unless providers rise to this task, outside groups, whether supporters or opponents, will define success and the lack of it for them.

And now, for the rest of the story...

Mend it, don't end it!

From Inside Higher Ed, about the recent NACIQI hearings on accreditation:

About two-thirds of the way through the first day of the Education Department's two-day forum on higher education accreditation, something strange happened: a new idea emerged.

Not that the conversation that preceded it was lacking in quality and thoughtfulness. The discussion about higher education's system of quality assurance included some of the sharper minds and best analysts around, and it unfolded at a level that was quite a bit higher than you'd find at, say, the typical Congressional hearing.

And now, for the rest of the story...

Accreditation reform: Dull but important!

Kevin Carey, who writes for Education Sector and lots of other publications, recently testified before the National Advisory Committee on Institutional Quality and Integrity

To be blunt, I believe the accreditors overseen by NACIQI should not be in the business of deciding whether for-profit colleges and universities should have access to the federal Title IV student aid system. The heart of accreditation is peer review. And the power of peer review does not lie with the creation of or adherence to specific rules and guidelines. Instead, it lies with shared norms and values. That’s all that “peer” means, in the end — persons or organizations with whom one shares fundamental ideas about the nature of things. Peer approval is extremely important and influential, as we see in the scholarly communities that thrive in higher education. But it doesn’t work if those being reviewed are not actually peers.

And now, for the rest of the story...

Adrift?

Robert Sternberg is a highly respected psychologist and, more recently, university administrator. He is probably best known for the notion of "multiple intelligences," which clearly informs this commentary.

This nation made a serious mistake in introducing well-intentioned but poorly executed legislation, the No Child Left Behind Act, which has turned many of our elementary and secondary schools into glorified test-preparation centers. Do we dare now do the same for colleges? Do we really want to make preparation for narrowly conceived cognitive tests the primary goal of a college education? Or do we want to broaden assessments, such as performances and portfolios, perhaps in addition to the narrower assessments? If we limit ourselves to narrow measures, we can say good-bye to our hopes to develop an internationally competitive, creative and ethical society. We instead can say hello to creating a nation of excellent test-takers who will shine, but only in some dystopian world in which achieving high scores on tests is the measure of one’s contribution to society.

Ultimately, the goal of college education is to produce the active citizens and positive leaders of tomorrow — people who will make the world a better place. Narrow tests of cognitive skills do not measure the creative, practical, and wisdom-based and ethical skills that leaders need to succeed. We can and truly must assess much more broadly.

And now, for the rest of the story...