Can We Gauge Technology's Impact on Learning Outcomes?

In a Campus Technology article, Ruth Reynard explores the questions of how instructional technology affects students’ mastery of learning outcomes and how the presence or absence of outcomes related to that technology affect the assessment of more traditional outcomes. Here are some excerpts:

The ongoing debate as to the effectiveness of technology use for student learning outcomes still seems to have no clear answers. Recently, some universities have decided to end their laptop programs for students because of the economic challenges facing those institutions. But there is no consistent response as to the effect on students. Some say it has been highly effective for students, and others say that it has not had any significant impact in how students learn.

What is interesting is that there is also no real agreement as to what should be measured or even whether it can be measured in order to quantify success in this regard. Institutions…that have adopted technology for instruction, often have little or no systematic methodology in place for instructional technology use or how its success can or should be measured. Rather, the technology use has typically relied upon individual teachers and faculty who have given up time to learn and use new technology and who are always underfunded and unable, as a result, to expand their use to other programs and other instructors for ongoing research.

What…can be done to truly assess benefits to learning in regards to technology use?

…much of the reason we are not discovering…any meaningful way of measuring whether technology truly improves the learning experience for students or helps them attain the learning outcomes more efficiently is that the knowledge of the technology itself is smattered and that little is consistently taught to those who are currently teaching. This is heightened with the move away from separate instructional or educational technology departments toward all-inclusive IT departments that, owing to budget constraints, house only one or two instructional staff. No self-respecting teaching "expert" will approach an IT help desk with a question like, "Could you please explain to me what a server is and what happens to my documents when I hit the ‘save’ button?"

Most of this, in my opinion, results from education itself being the commodity it now is and teachers being required, as a result, to be business-minded, currently marketable, and technology-savvy. The swing away from any or all of these by teachers, then, is more about defensive practice than retaining rigor.

And now, for the rest of the story...

That critical thinking you’re always hearing about…

In the March 26 New York Times, Nicholas Kristof asks:

Ever wonder how financial experts could lead the world over the economic cliff?

One explanation is that so-called experts turn out to be, in many situations, a stunningly poor source of expertise. There’s evidence that what matters in making a sound forecast or decision isn’t so much knowledge or experience as good judgment - or, to be more precise, the way a person’s mind works.

[Philip Tetlock, Berkeley professor, author of Expert Political Judgment, and “expert on experts”] called experts such as [those who’ve led the world over the economic cliff] the “hedgehogs,” after a famous distinction by the late Sir Isaiah Berlin…between hedgehogs and foxes. Hedgehogs tend to have a focused worldview, an ideological leaning, strong convictions; foxes are more cautious, more centrist, more likely to adjust their views, more pragmatic, more prone to self-doubt, more inclined to see complexity and nuance. And it turns out that while foxes don’t give great sound-bites, they are far more likely to get things right.

And now, for the rest of the story...

Is anyone closing the loop?

A subscriber to the ASSESS listserv (a wonderful resource, BTW, that I need to list over on the left side of this page) recently asked for examples of completed assessment plans from college or university departments. She wrote:

I can locate mission statements, goals, and student learning outcomes for a large number of departments; however, I am struggling to find the "Results" and "Continuous Improvement" examples.

I'll just add that it's also pretty easy to locate assessment data, findings, or results from deploying various means of assessment. But what's scarce as hens' teeth are examples of actions that programs have taken to improve their students' learning, based on assessment results.

Well, Jessica Johnson, Director of Institutional Assessment at the University of Nebraska-Lincoln, replied with the link to Exemplars: How is the Outcomes Assessment Process Contributing to Improvements?, summarizing how the assessment loop is being closed in some programs at UNL.

There was also a reply from Dave Eubanks, whose blog is listed over there on the left. What he says contains a lot of interesting and useful information as well as links to still other interesting and useful information. One of his links takes us to some completed assessment reports (some sensitive info redacted) from programs at Coker College.

Please take a look at these. For almost all of us the question, "What are you going to do about it?" has proven the toughest to answer. Examples of how others have answered this question in their programs is really valuable.

As always, there's considerable variety among the "actions based on assessment findings" that we see in Jessica's and Dave's (or anyone's) examples. What often strikes me in documents like these is the vagueness or lack of specificity about precisely what actions have or will be taken. For instance, saying something like "as a result of these findings, faculty now work more extensively with students writing their research papers to encourage more analytical thinking" doesn't tell us much.

Just for fun. Skip this one if you're in a serious mood.

Do any of you watch the HBO series, Flight of the Conchords? I have to admit I'm a sucker for good old-fashioned silliness, so I really like this show. And this song/video just knocked me out! I could watch it all day. I can't help it; so just go ahead & think what you will about me & my artistic (?) preferences.

Assessing how students learn

Here are some excerpts from Bill Cerbin’s article, “Assessing How Students Learn,” from the web site of the Carnegie Foundation for the Advancement of Teaching:

By measuring what students learn educators can monitor student progress, determine learning gaps and gains, and document achievement.

But measuring what students learn is of limited use if our goal is to improve their future performance. It is akin to taking a person's temperature. You may learn the individual has a fever but the measurement produces no insight into the cause.

To reduce the guesswork we need assessment that reveals how students learn - how they interpret and make sense of the subject, where they stumble, what they do when they do not understand the material, how they respond to different instructional practices, and so on. Understanding the basis of student performance can help us identify appropriate teaching practices or approaches.

Encouraging teachers to assess student learning as it takes place in the classroom can help them answer questions about how and why the gaps exist. Assessing how students learn can lead to the kind of information we need to make decisions about how to improve teaching and learning.

A compelling example of this form of assessment is the Berkeley calculus project which took place more than 25 years ago…

And now, for the rest of the story...

Faculty role models

Engagement. It's one of those buzzwords that I'd prefer not to use if I could find an effective alternative. But I haven't yet. Help, somebody!

In the meantime...

Maybe one of the experiences that best gets students engaged in their learning is seeing role models who have succeeded at what those students are attempting. Here's an article from the March 25 Chronicle of Higher Education about the profound effect that role models might have for our students. This article is mostly about graduate students and their faculty role models. But it seems to me that pretty much everything that's true for grad students in this story goes for undergrads as well. What do you think?

Eliminating versus: Education for life is education for making a living.

Some of you have read Rob Jenkins’ essay in the March 27 issue of the Chronicle of Higher Education entitled Our Students Need More Practice in Actual Thinking. Jenkins advances the position that what’s variously called “general education” or “liberal arts education,” which is housed in many institutions’ core curricula, really is relevant to students’ academic goals and “ultimately to their future professional lives.” The proposition that liberal education constitutes more than learning for its own sake, that it really does provide the essential preparation that all citizens require in order to lead lives that are personally fulfilling and beneficial to others, has always gotten lots of lip service.

But increasingly notable trends in higher education promise to back up that lip service with evidence. On the forefront of this movement is the Association of American Colleges and Universities (AAC&U). In the most recent issue of their publication, Liberal Education, Academic Editor, David Tritelli, writes that

…what’s distinctive about [AAC&U’s] vision is that, in foregrounding the broad consensus in favor of liberal education outcomes, it demonstrates that while liberal education remains the best preparation for life, it is now widely recognized as the best preparation for making a living as well.

The emphasis on the word, “demonstrates,” is mine. AAC&U has fostered considerable research over the years (and it continues) that has resulted in evidence to support their claims.

It’s hard to single out particular items to recommend for your reading because there are so many. So I’ll stick with that same issue of Liberal Education and ask that you read Debra Humphreys’ article, College Outcomes for Work, Life, and Citizenship. Following is a quote from the article. She makes frequent reference to LEAP, which is one of AAC&U’s primary initiatives entitled “Liberal Education and America’s Promise.”

…the LEAP vision updates as well as demonstrably builds on the enduring aims of liberal education: broad knowledge, strong intellectual skills, and a grounded sense of ethical and civic responsibility. But LEAP also moves beyond the traditional limits of liberal…education – moving, most notably, away from the self-imposed “nonvocational” identity and rejecting the more recent association of liberal education with learning “for its own sake” alone, rather than for its practical value in real-world contexts. The LEAP vision for student learning places stronger emphasis on global and intercultural learning, technological sophistication, collaborative problem solving, transferable skills, and real-world applications – both civic and job-related. As AAC&U’s president recently noted, “in all these emphases, LEAP repositions liberal education, no longer as an option for the fortunate few, but rather as the most practical and powerful preparation for “success” in all its real-world meanings: economic, societal, and personal.

Maybe all we need to do is work harder (?)

This article in Slate is about the Knowledge is Power Program (KIPP), which operates several charter schools around the country with unusual success among our least advantaged students. Because it started in my hometown and because its focus is on students like so many at my university, this caught my eye. I wonder if they're doing anything that could be successfully translated into a higher education setting.

It’s not all about money. However…

Here are some highlights from a March 13, 2009 Chronicle of Higher Education article entitled On the Bottom Line, Good Teaching Tops Good Research. My university is not a research institution like the author’s. But many of my colleagues would like to see us move more in that direction and, in fact, some movement of that sort seems to be happening. Though the numbers cited below would be different for my university, it would be interesting to do the math.

In part, research is more prized because it is easy to determine the dollar value that professors bring to universities through outside grants. It is more difficult to establish the economic value of good teaching…. But it is not impossible.

Our research-grant…income has increased, but our tuition income has increased much more. In 2008 grant recipients…brought in…$60-million, of which roughly $20-million was gross overhead - money given to the college to support the grant. But the research enterprise is expensive….By the time various administrative offices dip…into this overhead pool…$15-million is left to pay the electric bill.

During the same year, gross income from tuition was…$190-million…nine times as much as gross overhead income. Clearly we are today a tuition-driven institution, and while research supported by outside sources provides valuable intangibles…in terms of net income to the university, it's chump change.

So where does good teaching come in? The proportion of freshmen who return for their sophomore year…is 81 percent…. [T]hat means about 500 students a year don't return.

Because each student who did not return would have provided about $50,000 in tuition and fees over the three remaining years until graduation…that retention rate represents a loss of $25-million a year. That's $5-million more than our gross overhead income.

Some of these nonreturning students are not "recoverable…." But a substantial number are on the fence. They could stay, but maybe they will, and maybe they won't. What causes a student to stay? Twenty years of research on the "retention issue" can be summarized in one word: engagement. If the student somehow forms a bond with the institution, he or she comes back to the fold.

How is such a bond formed? It could be social…. Professors have little…to do with those things. But on the academic side, the biggest single factor seems to be: Does a member of the faculty care whether the student lives or dies? How can a student possibly have that feeling when he or she is part of an introductory sociology class of 300? Not easily - but with an investment of time and effort, it can be done.

I teach 600 freshmen a year in my two big introduction-to-biology classes…and a number of years ago, the failure rate started to skyrocket…. So I decided to see if I could do anything that would have an impact….

I announced to my classes that those who failed the first exam would be required to come in and see me for a diagnostic interview. I have had over 95-percent compliance. How? I simply suggest that the result of their not coming in will be so dreadful that I can't even mention it in class. Power of suggestion does the rest. Because more than 100 students typically fail the first exam, I essentially do nothing but see students for two weeks after that. Although they're a bit scared at first, most of them say "thank you" after the interview is over. The first semester that I tried this, the failure rate dropped by 40 percent. I continue the practice to this day with excellent results.

Of course, all the time I spend with these students I could be working on grant proposals. However, out of my 600 students, 114 are statistically at risk of not returning. If, through this personal attention, I "salvage" only five of those students, I will have recovered $250,000 in lost tuition. And I can do that every year. In my discipline, that is far more than I would ever be able to generate in grant overhead.

My university has 27 departments that have an average of at least two courses with more than 150 students each. If instructors in each of those courses "salvaged" the same number of students in their courses as I am sure I do in mine, almost $15-million in otherwise lost tuition could be recovered - about the same amount generated by grant overhead minus expenses.

Can faculty members be trained to be more effective teachers and so have an impact on retention? Absolutely. Instructional-development programs traditionally do just that. These offices are typically marginalized and token at research universities, without appropriate money, prestige, or appreciation. Faculty members typically have no official incentive to seek advanced training in teaching; in fact, they are often discouraged because of the disproportionate emphasis placed on research "productivity."

If research universities like mine, facing hard economic times, are serious about improving their bottom lines, they need to improve teaching. That goal, with the potential to yield far more income than grant overhead, deserves an appropriate investment in time, money, and recognition.

Service learning: How to do it right

The first set of quotes that follow are from High-Impact Educational Practices, authored by George Kuh in collaboration with the Association of American Colleges and Universities ( AAC&U). In his work with AAC&U, Kuh helped...

...spotlight and verify a set of “effective educational practices” that, according to a growing array of research studies, are correlated with positive educational results for students from widely varying backgrounds.

Now drawing on new research, Kuh...shows that the practices...initially described as "effective" can now be appropriately labeled "high-impact" because of the substantial educational benefits they provide to students.

The results of participating in these high-impact practices, Kuh shows, are especially striking for students who start further behind in terms of their entering academic test scores. The benefits are similarly positive for students from communities that historically have been underserved in higher education.

One of the high-impact practices identified and verified by Kuh is service learning (AKA community-based learning). According to Kuh…

…in these programs, field-based “experiential learning” with community partners is an instructional strategy – and often a required part of the course. The idea is to give students direct experience with issues they are studying in the curriculum and with ongoing efforts to analyze and solve problems in their community. A key element in these programs is the opportunity students have to both apply what they are learning in real-world settings and reflect in a classroom setting on their service experiences. These programs model the idea that giving something back to the community is an important college outcome, and that working with community partners is good preparation for citizenship, work, and life.

One of the reasons why some of us tend to be skeptical about claims like these is that apparently good ideas about improving student learning have always existed in abundance and have often been tried, but with few, if any, beneficial results. And says Kuh, “While high-impact activities are appealing for the reasons…outlined, to engage students at high levels, these practices must be done well.”

OK, so how do you do service learning well? Across the board, colleges and universities struggle with service learning's twin goals of providing meaningful help to the community and academic rigor to students. The next set of quotes comes from a Feb. 27, 2009 Chronicle of Higher Education article.

Many fads came and went in the 1980s. But the idea of linking service and learning never went away.

The concept was kick-started in 1985, when several college presidents decided to counter the popular image of college students as disaffected, materialistic, and self-absorbed. That year the presidents of Brown, Georgetown, and Stanford Universities started Campus Compact….

The nonprofit group embraced the service-learning model, and by 2003 its membership had grown to include a quarter of all colleges in the United States. Students were logging millions of hours each year. But around that time, leaders in the field began to ask, "Service to what end?"

…colleges began to take a closer look not only at learning outcomes but also at whether students' volunteer work was actually making a difference in their communities. On both counts…many colleges were falling short.

That is something even colleges on the leading edge of service learning still worry about. California State University at Fresno…sends more than 10,000 students into the community each year…. But sheer numbers aren't enough….

The challenge is to focus not only on the quantity of volunteers but also on the quality of the work they are doing. At Wagner College, on Staten Island, the provost dreamed up an idea to do just that.

The approach…connects entire academic departments with single agencies on the island. That setup enables the college to draw up a financing proposal for a soup kitchen rather than just serve meals. And faculty members and their partner agencies develop syllabi for courses together - ensuring a clear connection between the soup line and the sociology tome on poverty.

As at Wagner, [CSU-Fresno] plans to start directing those volunteers to one particular neighborhood, in West Fresno, to maximize their impact. It is a move more colleges need to consider….

"One of the risks is that a lot of schools think, 'Oh, well, we're just going to do something,'" he says. "My fear is that people rush into this stuff, as they see donations and press, and that they don't really think about, 'What are the needs in the community?'"

And now, for the rest of the story...

Six ways to keep students beyond the first few weeks

Here’s the Chronicle of Higher Education’s take on Imagine Success: Engaging Entering Students, the just-released report on findings from the Survey of Entering Student Engagement, which is mentioned in an earlier post on this blog.

Interview with director of AAC&U's VALUE project

Some quotes from Terrel Rhodes:

...the VALUE Project (Valid Assessment of Learning in Undergraduate Education) works to develop approaches to assessment based upon examples of student work completed in their courses and saved over time in an e-portfolio. The project collects and synthesizes best practices in assessing student work using rubrics developed by faculty members. One of the project’s core purposes is to identify commonalities of outcome expectations of achievement across a variety of institutions.

It is ironic that just at the point when higher education research has finally developed a rich information base on effective practices that enhance learning, on cognitive development and neurobiologic bases of knowing, and technological advances that greatly expand our abilities to collect, preserve and demonstrate complex, multi-faceted learning, that we so willingly accept outmoded, snapshot, shorthand representations of the value of our educational outcomes and impact on student learning.

In contrast, the VALUE project responds to the need for multiple measures of multiple abilities and skills, many of which are not particularly well suited to snapshot standardized tests. The types of learning that employers and policy makers are calling for need to be demonstrated through cumulative, progressive work students perform as they move through their educational pathways to graduation; rich, multifaceted representations of learning in curricular and co-curricular contexts, rather than artificial examinations divorced from applied contexts.

We hope that the VALUE project will be able to demonstrate several things: that faculty across the country share fundamental expectations about student learning on all of the Essential Learning Outcomes deemed critical for student success in the 21st century; that rubrics can articulate these shared expectations; that the shared rubrics can be used and modified locally to reflect campus culture within this national conversation; and that the actual work of students should be the basis for assessing student learning and can more appropriately represent an institution’s learning results.

Our experience at AAC&U in working with faculty on campuses across the country is that faculty are typically eager to have permission to talk about and to focus on student learning. Once you get beyond complaints about teaching is not rewarded adequately, etc., faculty embrace discussing learning and teaching. So, there is no difficulty in getting faculty interested in talking about the subject. The biggest barrier is often a lack of awareness about options for assessing learning and what it would take for the individual faculty member to adapt what they know and are familiar with to some new environment or process.

Increasingly, the investment in e-portfolios is becoming less and less of an obstacle for campuses since there are free Web tools that students can use to construct e-portfolios.

Having been a faculty member on several campuses for over twenty years, I know that using rubrics and e-portfolios does not have to create more work--it requires working differently, shifting my time and focus a bit--but it is richer and more rewarding than what I used to struggle with in trying to communicate my expectations for learning and how students could more readily succeed in meeting those expectations. There is a transparency and communication ability that enriches the conversations both with students and with colleagues.

And now, for the rest of the story...

Growing interest in portfolios as approach to meaningful assessment

In yesterday's Inside Higher Ed, David Scobey noted that...

We humanists are notoriously hostile to systems of assessment. We tend to believe that the most important effects of a humanities education resist measurement: nuanced communication skills, reflective dialogue between theory and interpretation, attention to context and complexity. Conversely the outcomes that can be most readily measured seem to us the least salient: informational content in a sub-discipline, performance of competent analyses according to check-listed rubrics.

He went on, though, to suggest ways in which the educational outcomes most valued by humanists (and not just humanists, I'd like to add) can, in fact, be assessed satisfactorily. Not surprisingly, the approach he suggests involves portfolios. The article is well worth reading for anyone who's interested in assessment of student learning outcomes that's really meaningful rather than just going through the motions. By the way, be sure to read the very interesting comments posted by readers at the end of the main article.

Now comes today's article in the Chronicle of Higher Education which quotes still another article in Academic Commons: “ePortfolios may be the most likely vehicle to help us make the transition to an academy of the future that is both relevant and authoritative.” The Chronicle article is organized around the following four main points:

• First, ePortfolios can integrate student learning in an expanded range of media, literacies, and viable intellectual work.
• Second, ePortfolios enable students to link together diverse parts of their learning including the formal and informal curriculum.
• Third, ePortfolios engage students with their learning.
• Fourth, ePortfolios offer colleges a meaningful mechanism for accessing and organizing the evidence of student learning.

The article also contains links to several valuable resources, including web sites of institutions that are leading the way in the use of ePortfolios.

Scholarly output rises, undergraduates are disengaged.

“I think these two trends - to do more and more research and less academic engagement on the freshman level - are not unrelated,” Bauerlein said in an interview about Professors on the Production Line, Students on their Own. The American Enterprise Institute for Public Policy Research released the paper Tuesday.

And now, for the rest of the story...

Is the best chance for student engagement in the classroom?

Over the past month I've been involved in a series of small group discussions that constitute part of a "dialogue-to-action" project aimed at stimulating broad community support for our Achieving the Dream initiative. For me, one result of these conversations is the strengthening conviction that one of the best things we could do to improve student retention and academic success at our 100%-commuter university is to build up our student life (AKA student affairs, student activities) programs and incentives so as to promote more and stronger connections among students, students and faculty, students and staff, and all the other combinations. Engagement. We run into forms of that word, and the collection of concepts they stand for, at every turn these days, so much so that I worry if those words might become meaningless due to overuse.

Anyway, building up student life programs is a daunting challenge, especially at schools like ours. We certainly should continue doing all we can to address that challenge. But because I've been thinking a lot about promoting student engagement via student life programs lately, it really caught my eye the other day, and caused one of those "Well, duh!" moments when I read that "given most students' limited time on campus, the best opportunities to build relationships often are found in engaged learning (both in and out of the classroom) and other structured experiences." That's from Imagine Success: Engaging Entering Students, a report from the Center for Community College Student Engagement. This week several stories have appeared in the higher education press about the report, including this one from Inside Higher Ed.

Of course "engaged learning" can take many forms. And despite what we sometimes seem to be hearing, it can even happen in the context of the classroom lecture, as Rob Weir discusses in another article from Inside Higher Ed that appeared today.