Reading the Foreword to "Ungrading"

Following my "Experiments in Grading" post of a few weeks ago, a colleague suggested that I peruse the newish volume Ungrading (ed. Susan Blum; West Virginia University Press, 2020). I haven't yet gotten far in the collection, and in any case I don't expect to read the book cover-to-cover, since not all of the topics are equally applicable to my interests or pedagogical problems. However, I had some reactions while reading the Foreword by Alfie Kohn. I'll record a few of these thoughts here, and follow up with future posts as I read more of the book. (I'll also continue to edit and expand my earlier post on grading.)

In general, in the Foreword (and, from what I can tell, perhaps the rest of the volume as well), the case against grades is vastly overstated. Kohn describes nine steps along the path from grading to ungrading--and although I have followed some parts of this path myself, my underlying rationale is almost always very different from the reasons stated here.

Kohn's first step:

We start by worrying about grade inflation before gradually coming to realize the real problem is grades themselves.

I certainly have worried at times about grade inflation...but what about the second part of that sentence? I would frame the issue not as "the real problem is grades themselves," but rather as "grades are problematic." In my previous post, I outlined a number of reasons why I think that grades are problematic, mostly involving the conceptual overburdening of grades, and attendant practical issues that arise as a result. I stand by this, yet it doesn't necessarily follow that grades themselves are "the problem." Grades do accomplish at least some useful purposes, including signaling. Indeed, if Bryan Caplan is correct that signaling is the overarching aim of academia and schooling, then some form of grades are inevitable, since the system wouldn't be able to function unless something facilitated this basic end. Of course, as I also acknowledge elsewhere, to the extent that grades serve as signals, they are imperfect signals, not least because they don't usually carry any explanatory detail. To outsiders who want to know whether a student is competent, grades only indicate the barest outlines of an answer; and to students receiving grades without an accompanying explanation, no real learning can take place because the feedback implied by grades is too diffuse to provoke any particular improvements. But, be that as it may, some version of this problem will be replicated in virtually any system for signaling quality that goes short of actually describing, in real detail, individual skill-sets. For this reason, some system like grades is probably a necessary part of life. It isn't clear that this system or any other is itself "the problem".

Kohn's second step:

...We make sure that everyone can, in theory, get an A. Only then do we realize that rating, too, is a problem, even if a less egregious one than ranking. We've eliminated the strychnine of competition, but there is more to be done if we're still dispensing the arsenic of extrinsic motivation.

This is an extension of the previous point: ratings are inevitable. But, I think it must be added, they are in some cases beneficial. The only way to improve is to receive feedback on one's abilities and performance, and although I agree that grades aren't the best way to deliver feedback (an objection I raised earlier), it isn't the fact of rating that is the problem, but the fact that the system of rating involving grades ends up, in its purest form, delivering ratings without explanations. The solution here could be very robust rubrics, or detailed commentary. About extrinsic motivation, this doesn't strike me as being obviously problematic. Students are individuals, and as such will be motivated by any number of different considerations; who am I to tell them that only intrinsic motivations matter?

Kohn's third step:

...We stop using letters and numbers to rate what students have done and instead use descriptive labels such as "needs improvement," "developing," "meeting/exceeding expectations," "proficient," and "mastery." Step two: we realize these labels are just grades...by a different name and that we need to get rid of them too.

This is the first point at which my disagreement becomes serious. Such labels are not grades by a different name; they are partial explanations of why a student has succeeded or failed at a particular task, and in what ways. Unlike a straight-up letter grade delivered without accompanying commentary, these explanations do, or at least can, make clear what needs work, and what kind of work would be best. They aren't perfect, of course; and crucially, they leave room for imprecision that can, in the wrong pedagogical hands, end up being just as useless to students as grades themselves often are. But what would remain of education if we got rid of the possibility of telling a student that work needs improvement?

(Incidentally, I also find labels such as these useful when they are applied by editors to my own professional work. One of my own intuitive measures of how useful a particular comment might be to a student is my sense of how helpful the comment would be if an editor voiced it while working through one of my manuscripts. I want to know what doesn't work, what could be better, and, ideally, what steps I might take to make the necessary improvements. Likewise, I assume that students at least want to have the option of hearing similar thoughts from me--and these comments are a baby-step in the right direction, even if they do not end up saying enough.)

I agreed with Kohn's fourth through seventh steps, which advocate for more precision and description in feedback to students--though, again, not because I think grades themselves are bad (which is Kohn's ultimate point) but because I think they are often overburdened or poorly applied, and one of the ways to remedy this is to be more explicit about what the grades are supposed to accomplish.

However, in step 8 we encounter "ungrading" in its most usual form: "we meet with students individually and ask them to propose course grades for themselves, while reserving the right to accept their suggestions." Step 9 is simply an intensified version of step 8: students choose their own final grades, and we reserve no right of refusal.

One problem with this approach is that it places undue psychological pressure on the students. Students find themselves in the position of second-guessing what we professors think about them, and trying to weigh this against what they then think we might think when we see the grade they propose...and I imagine that such mental calculations are ultimately pretty harmful--more harmful, certainly, than what I think they experience when I just go ahead and give them a (probably decent) grade along with some constructive feedback.

Another problem is an extension of what I have already mentioned. Grades do serve a purpose in the culture as a whole (even if mainly for outward-facing signaling), and to strip them of that purpose--which would surely occur if every prof left grading entirely in the hands of students, since external bodies would no longer think of grades as a reliably "objective" standard--would most likely lead not to the abandonment of rating systems altogether, but simply to a new kind of rating system: a different version of the same thing...and here I would invoke the classic argument from conservatism. Why fly to ills we know not of? When you don't yet know what the new system will be like or what further problems it may bring, it's important to make changes that are cautiously incremental rather than radical. It should follow that the best way to deal with the problems posed by grading as it's currently practiced is not to tinker with the culturally-shared elements of the grading system, but to experimentally tweak those aspects that can be safely adjusted within the comparatively private context of one's own syllabuses.

The post formerly known as: Is "Bad Music Love" Equivalent to "Bad Movie Love"?

In mid-January 2022, I wrote a somewhat informal blog-review of Matthew Strohl's excellent, recent book, Why It's OK to Love Bad Movies. As it happens, I ended up reviewing the book a few months later for the Journal of Aesthetics and Art Criticism - and once I had signed the journal's publishing agreement, I needed to remove the blog post, which overlaps a great deal with my review. When the review is officially published, I will place a link on this page; however, in the meantime, I have deleted any passages that made it into my review...and am leaving whatever passages are unique to the blog post. Although the material that remains is probably fairly impossible to understand, I wanted to preserve these thoughts, since they seem interesting and relevant to other ideas (even if they remain in a somewhat incomprehensible state when isolated like this).

---

The best book I have read so far this year is Matthew Strohl's recently released Why It's OK to Love Bad Movies. Granted, the year has only just begun; yet I think this is probably the best book on the philosophy of art I've read in a long time. The writing is spectacular; the arguments are convincing; and most of all, the sheer love of film conveyed throughout the book is wonderful. Strohl accomplishes in a breezy 194 pages what many other philosophers require many hundreds of pages to do.

Strohl's book is ostensibly about movies; but in fact, it is a defense of the good life. The argument he builds is, initially, focused only on film. He begins by defining a stance of aesthetic appreciation for bad movies, termed Bad Movie Love. It goes like this. When saying that a movie is "so bad it's good,"

"'Good' is being used in the final sense while 'bad' has a special meaning. ...One recognizes that there is some limited sense in which the movie is bad, but...one ultimately judges it to be aesthetically valuable, in part because it's bad in this limited sense." (p.4)

This is a fine place to start, though at first the idea seems like it might veer into question-begging, since even if a bad movie is judged aesthetically valuable, it isn't initially clear why we'd want to dwell on bad movies when there are so many good movies around. I agree with Strohl that Batman and Robin is "so bad it's good"; but for a decent portion of the book I don't yet see why such a film is more worthy of my time than Vertigo, which, let's face it, is "so good it's good". (Or In the Mood for Love, or Adaptation, or Mad Men, or any of the other good-good things I've watched or re-watched recently.)

....

I'm fully convinced by this. I now feel personally liberated to enjoy some of my guilty-pleasure films. Indeed, I'm not only convinced by the arguments, but inspired by them.

It's natural to ask, though, particularly in light of my other interests, whether these arguments apply equally to other areas of aesthetic pursuit. Being both a musician and an avid reader, I can't help but wonder whether Bad Music Love (or Bad Novel Love) is as permissible as Bad Movie Love.

My guess is that, at least in the case of music, the answer is: no. Bad Music Love might be far, far worse than Bad Movie Love. Why? First, the disclaimer: I don't think that I'm biased by being a musicologist. True, my professional work demands that I have discerning musical tastes, and do I spend a lot of time trying to articulate why various compositions are good or bad. And I recognize that my general disposition may seem to place me in the same category as Strohl's imaginary "Professor Stuffypants". But I don't think these biases impinge on my reasoning about this particular point (though my friends might say otherwise...).

My hypothesis involves both the relative quantity of bad music vs. bad movies, and the relative quality of bad music vs. bad movies. I suspect that there is far more bad music in the world than there are bad movies, and that the bad music is infinitely badder than the bad movies.

...

I could write a bad piano sonata today, alone in my apartment, with no money or resources beyond some paper and a pencil. (I don't even need an eraser! This is supposed to be bad; why bother revising?) I would have a harder time making a bad movie. Perhaps I could pull it off with my phone and a selfie-stick...but this would not be the kind of bad movie Strohl would watch, since it would have no reliable distribution, and thus would be unlikely to make its way to his TV. This question of distribution raises another point: it isn't only that bad movies take more time and effort to make than bad music, but that their chances of being preserved and distributed is relatively low. Anyone can write bad music on a manuscript leaf and find that, a couple hundred years later, it will be digitized in the Duben Collection or in the Dresden State Library, awaiting discovery by zealous archivist period-performers who want to play bad music; but not so for bad movies.

The barriers to the distribution of bad movies are not the only salient considerations. Movies (even bad ones) require more people to make than music. This also has its effect. Except in some exceptional cases in which a bad filmmaker has unchecked power (cf. Strohl's discussion of Plan 9 from Outer Space), most movies are made with the involvement and input of more people. This alone almost guarantees the presence of error-correcting mechanisms, since even a bad director may collaborate with people who end up improving the final product. For this reason, I suspect that even most "bad movies" aren't really all that bad--a suspicion corroborated by Strohl's book, which argues that many of these movies, even the Twilight films, are actually rather good. Bad music, on the other hand, can be profoundly bad. Often, no external ear has been engaged to criticize and correct the final product; it is the composer's own intuitions, errors and all, that features in what we end up hearing. (Incidentally, this has changed in the modern era of pop music, where songs are very often written by a performer in collaboration with others behind the scenes...and it's probably for this reason that the percentage of competent pop songs I encounter is far higher than the percentage of competent non-Mozart/non-Haydn 18th-century symphonies I encounter.)

Does any of this change what I make of Strohl's arguments? I suspect so. Perhaps his defense of bad movies is not really a compelling defense of bad movies, but a defense of medium-bad or even pretty-good-but-not-great movies. The movies Strohl discusses are, ultimately still worth our time. We may not be in the mood to ingest Vertigo (just as we may not be in the mood to follow Bach fugues or parse a Schoenberg piano sonata); but in Strohl's hands many movies that seem superficially bad can still contribute much to our life. This is more than I would say of most musical compositions I encounter.

...

Perhaps the saving grace in the case of novels is that, as with film, books require the involvement of multiple people along the path from manuscript to published edition, and thus present opportunities for improvement and error-correction. Much of the art-music produced over the past 700 years survives only in manuscript, or in early published editions, and is thus hit-or-miss in a way that may not apply to published novels or professionally distributed films. Of course, according to Sturgeon's Law, most work in any given domain is bad--and this applies everywhere, including to published novels and professional films. But the error-correcting processes in these media nonetheless seem to function reasonably well in improving overall quality.

One final--contentious--possible explanation for the general high quality of film in comparison with older music and older texts is that aesthetic standards overall have improved over the past century, and thus that art-forms invented more recently have the benefit of having developed within a context of greater artistic and aesthetic understanding. The fact that composers like Bach, Mozart, or Beethoven figured out how to write reliably great music during the 18th century is, when you think about it, completely remarkable considering just how little progress had yet been made in many other areas of cultural, technological, scientific, and philosophical life. That they did succeed where so many of their contemporaries failed is testament both to whatever progress had occurred during and just before their lifetimes; and to the intellectual labor each one of these artists did in improving aesthetic knowledge. (There was probably also a lot of luck involved.) Yet as more general cultural and scientific knowledge improved, so too did artistic standards: thus, my experience suggests that there is a very high likelihood that an unknown, non-canonical piece of music written after around 1850 will be pretty decent; whereas I can't say the same for the period 1750-1800. That film was invented in the 20th century may be at least a partial explanation for the fact that, as Strohl shows, even bad movies can be aesthetically valuable.

Experiments in Grading

My transition into professional academia has, for the most part, been very smooth. The "three pillars" of academic work - research, teaching, and service - all involve activities that I fundamentally enjoy; and, as a result, my job requires of me few tasks that seem onerous. Yet one point of consistent confusion and perplexity during my first few years as a professor has been...grading.

Before going to college, I thought a lot about grades: not philosophically, but in the more mundane sense that I was always worried about what my transcript would look like, and what effect this might have on my future. When I was an undergrad, and I could exert a bit more control over the selection of courses in which I enrolled, my grades improved. I stopped worrying, and indeed I stopped thinking about grades altogether. This changed a bit when I started grad school in the UK, but only because the cultural practices of grading differ so much across the pond. I went from receiving routine A's to receiving routine 70s, and it took a bit of adjusting for me to realize that this wasn't actually a massive step down. I stayed in the UK for my PhD, and thoughts of grades once again vanished from my life.

But now, as a professor, I have to deal constantly with grades. I spend the beginning of each term instituting grading scales and points-schemes that (I hope) will incentivize students to participate in certain ways in my courses; I spend the bulk of each term experiencing in real-time the results of such schemes; I spend many hours as the term progresses assigning grades; and then I spend the final portion of each term calculating course grades and dealing with grade submission, extensions, and other administrative joys.

But in addition to the fact that I am forced to think about grades through constant exposure to activities related to grading, I am also inclined to think about grades for other reasons. The most pressing reason is ethical, as I consider my relationship with the students in my courses and the impact that my grading practices will have on their futures. Another reason, more practical, is that some casual relationship exists between the structures and grading schemes I institute and the experience of my courses--and I would like to make those course-experiences as positive as possible for everyone, including myself.

All of this has led me to experiment a fair amount with different styles of grading in my courses. Now, as the Winter 2022 semester approaches and I put the finishing touches on yet another syllabus, I decided to reflect about some of these experiments, from previous terms and the coming term.

Initial reflections: what do grades accomplish?

Although this is not the place for me to sketch a comprehensive theory of grading, there are a few premises on which all of my experiments depend. Some of these have little to do with grades as such: for instance, I believe that students are people, that they are as intelligent and rational as I am, and that they are the best judges--certainly better than I could be--of how their own time should be used. I also believe that students are individuals and that learning takes place through creative processes that occur within individual minds. These background beliefs, broadly libertarian, lead me to avoid exams or tests, or any other method of assessment that would require all of the students to do the same activity, answer the same questions, or "prove" that they've acquired the same bit of knowledge. Instead, I try to minimize course requirements and come up instead with assignments that offer the highest possible degree of flexibility. (This is delicate, because flexibility for the students often translates into administrative demands on me--and one corollary to my belief that the students are people is that professors, too, are people, and thus deserving of flexibility and freedom.)

Other premises are explicitly about grades themselves: for instance, that grading as it is generally practiced often seeks to accomplish multiple different (and often conflicting) aims, including 1) signaling students' level of accomplishment or competence to the outside world; 2) providing direct feedback to the students; 3) ranking students into hierarchies of achievement; and 4) enforcing disciplinary practices. It seems intuitively obvious to me that these four aims should not all be bundled together; thus, many of my experiments and explorations are aimed at un-bundling them, and attempting to find ways to accomplish each of those aims (to whatever extent I wish to accomplish them) more coherently, explicitly, and fairly.

Of the four uses of grades, the one in which I could find the least wiggle-room was (1) signaling students' accomplishments and competency to the outside world. Grading practices are culture-wide, and if I alone decided to use grades in a radically unique way, this would certainly be misunderstood by future admissions committees and outside evaluative bodies, and this would have negative consequences for my students. By contrast, (2) and (4) could be accomplished within my courses using different means (say, by giving substantive verbal feedback).

Undergraduate grading

Thus, my first order of business when I began to design courses was to lower the burden on grades. I began by removing disciplinary measures from my grading schemes. When I was a student, it was normal for late work to lose 10% of its mark for each day (or hour, or any other arbitrary unit of time) that passed after the due-date. Ditto for errors of spacing, formatting, font-size, and so forth. I did away with this, announcing in my syllabuses that grades would reflect only the content of assignments, not the circumstances of their submission.

This meant, in practice, that I had done away with my most effective means for enforcing deadlines. In all but one of my courses, this has been a success, and students did not take advantage of my flexibility. Of course, some students submitted their work very late--but allowance for late submissions is part of my being convinced that the students understand their own interests and schedules better than I do. This flexibility became a problem only in the large (60-student) course I taught in Fall 2021, when a very high volume of late submissions made it difficult for me and the TAs to keep up with grading. I will teach this course again in Fall 2022, and one solution may be to set a final deadline for late submissions: for instance, to say that I will not use grades to enforce deadlines, but that any student who wants late work to be accepted at all must make arrangements with me beforehand. This way I preserve the flexibility (since I would plan to accept all requests for late work) but would prevent students from taking advantage thoughtlessly.

Another experiment, instituted early on, was to do away with grades-as-feedback. In principle, this is simple: just write detailed feedback for each submission. In practice, it is more complex, because it leaves open the question of how to assign grades. My solution was to think of each assignment as being pass-fail (which, incidentally, is how things work in the real world of professional research, where articles either get published or they don't). I approached this (in a 30-student undergrad course) by assigning higher-than-I-would-otherwise-give marks to passable papers (everyone with a decent paper got a B+ or higher), writing extensive feedback to each student, and assigning lower-than-I-otherwise-would marks for bad papers, offering the authors a chance to revise and resubmit. This course, taught online in Fall 2020, was perhaps my most successful experiment in grading: the students uniformly stopped worrying about their grades and started thinking about their ideas--and the result was that the papers became more interesting, more daring, more fun to read, and better as the term went on. Although I had to do a bit of extra work when students resubmitted their papers, it was well worth it..and the number of revisions I requested decreased hugely over the term. (I will also add that for this course, the paper prompts were very open-ended, meaning the students were free to find ideas that interested them and write about this. This also made the papers more enjoyable to grade, since in the best cases it stimulated the students' creativity, and in the worst cases it offered a minimum of at least some variety among the submissions.)

This approach has been a success in approximately five out of the six courses in which I attempted it. The course in which it was least successful was, as mentioned previously, a 60-person course, in which the comparative flexibility resulted in some submission-related chaos as the term progressed. Another potential problem in that course was that the students were caught off-guard by my insistence upon letting them follow their own interests when crafting writing assignments. Some didn't understand how to select a topic, and were reluctant to ask for help; others used the lack of disciplinary grading as an excuse to ignore the deadlines and let work pile up over the term. Others, of course, thrived, and wrote a series of brilliant and incisive papers. I don't count this experiment as a complete failure, since many students did ultimately report that they enjoyed the course and the assignment structures; however, I will modify the policies before I teach this course again.

Grading in graduate seminars

As of January 2022, I've taught three graduate seminars as McGill, and will be teaching my fourth during the coming term. The challenges presented by grading in grad seminars are different from those I encountered while teaching undergrads. On the one hand, graduate students generally receive As in seminars (or at worst, A-); thus, grading is relatively unimportant as signal, feedback, or ranking. On the other hand, the point structures and nature of assignments in graduate seminars still influence the course experience, and in some cases matter even more than they do in undergraduate courses, since so much of the professional initiation that occurs in grad seminars depends on individual reading and writing practices.

I've generally been bolder about experimenting with grading systems in grad seminars than in undergrad courses: first, because, as I've already said, everyone ultimately gets an A, which means the stakes are lower if my experiment goes awry; and second, because if things do go awry, grad students are more likely than undergrads to be sympathetic and professionally interested when I lay out the rationale for my experimentation.

For the first two grad seminars I taught, my grading experimentation (such as it was) involved similar parameters to those used in my undergrad courses: I didn't use grades to enforce deadlines, but I nonetheless scheduled assignments, gave each a point-value, and demanded that the students complete the work for credit in the course.

Grading, scoring, and board games

In Winter 2021, however, I began to experiment more radically. For my seminar that term, I hit upon the idea (stimulated by reading Thi Nguyen's new book Games: Agency as Art) of developing a points-based grading system resembling the scoring systems of board games. In general, in many of the board games I enjoy, players win by accumulating points--but the specific ways in which they accumulate points are left largely up to them. In the Ticket to Ride games (favorites while I was a PhD student), you can accumulate points by completing "routes", or by laying track aimlessly, or by building stations, etc. Each player works largely independently (though within a structure that emerges as a result of the other players' actions), accumulating points in any way and at any pace that seems advantageous.

I tried applying this ideal to my course. Rather than establish a series of required assignments, I offered a menu of possible assignments, each of which was attached to a point value. Give an in-class presentation for 3 points; write a short argumentative opinion piece and distribute it to the class for 4 points; write a response to an in-class colleague for 4 points; write a term-paper for 10 points; and so forth. I imposed no deadlines, but simply told the students that they needed to accumulate 29 points by the end of the term in order to receive an A in the course. They could write three term papers; they could write one term paper and give seven in-class presentations; etc. I created a simple spreadsheet to keep track of the students' scores as they wracked up points...and the course's grading scheme became a board game.

As with any radical experiment within a course, the results were very mixed. Some students thrived on the points system, since they were able to organize their time more thoughtfully (for instance, they could sign up for presentations or submit written work during weeks when they didn't have lots of work to do for their other seminars where stricter deadlines were imposed). For other students, the points system provided too much freedom, and left them feeling unmotivated, since they didn't have deadlines to work towards. I have not yet hit upon a solution to this problem. It is not clear whether the right approach is to try the board-game system again, and simply present it in a different way at the beginning of the term, to prepare students to succeed with it, or whether the system itself must be tweaked to provide a slightly more rigid framework.

Winter 2022: writing, research, and teaching skills

For the coming term, I have split the coursework into two different categories. One category, involving in-class presentations, responses, and other standalone assignments, will follow the board-game system, with no due dates, and the expectation that the students will accumulate some quantity of points by the end of the term. (More about this category below.) The other category, culminating in the submission of a term-paper, will introduce a new system, inspired by the writing habits that I hope to help the students develop. Rather than leave a term paper until the end of the semester, I have broken down the paper into individual components (thesis statements, outline, introductory section, body paragraphs, etc), which the students will complete progressively as the term unfolds.

By distributing this writing project over the entire term, I hope to both instill good writing habits in the participants and give them frequent but low-stakes goals to meet, so that the lack of deadlines in the board-game components of the course will not be demoralizing. I hope, if all goes well, that this will help to quell my own existential confusion about the nature of grad seminars and what exactly students are meant to take away from these courses. Having spent my entire graduate-student career in the UK, I never took a seminar during my PhD studies; thus, I lack recent models for how such courses can be taught and what they are intended to convey. In my other seminars, the discussions have been interesting (to me, at least), but I've never quite known why we should discuss these topics, particularly when the topics or the reading lists follow my own idiosyncratic interests rather than the research programs of the students. I recognize that there is an element of intellectual apprenticeship to the grad seminar: the students come to understand how I see my way through an academic issue, or how I compile a reading list, and perhaps, through a bit of effort along the way, might develop skills resembling mine.

In the current term's seminar, the freestanding assignments involving presentations and readings have also been tweaked slightly in the hopes of increasing their practical utility. The writing assignment is designed to instill good habits; but I have angled the other projects, too, towards professional skills. Rather than coming up with an exhaustive reading list, I have left open slots on the reading list which the students themselves can (for points, of course) fill in. If all goes well, this will give them a chance to exercise the skills of hunting down sources and navigating the dual constraints of finding sources that match their own interests and that match other scholars' (i.e. their classmates') interests. The in-class presentations, too, will not be devoted simply to the summarizing of texts and analysis of arguments, but to presentations of the kind they might someday give in an upper-level undergraduate course. This will allow them to test out pedagogical styles while also engaging with a range of scholarly sources.

I will continue to reflect on this system's success (and the challenges it presents) as the term continues! One major variable is the fact that, owing to the new Covid surge, the first few class sessions will be moved online. My Winter 2021 seminar was held over Zoom, and I found the online format to be a challenge, given that it sometimes impedes free-flowing discussion.