Click to Skip Ad
Closing in...

Criticwire Survey: The Rating Game

Features
May 12, 2014 9:36 AM
2 Comments
  • |

Every week, the Criticwire Survey asks film and TV critics two questions and publishes the results on Monday morning. (The answer to the second, "What is the best film in theaters right now?" can be found at the end of this post.) Send suggestions for future questions to sam at indiewire dot com.

Q: In his response to Vocativ's ranking of critics by how often they agree with a movie's Metacritic average, "most reliable" critic J.R. Jones takes exception to the practice of affixing letter grades or a star rating to a review. How do you feel about placing movies on a scale? Would you give your reviews a grade if you were calling the shots? And when you write for publications that require them, how much thought goes into choosing the rating versus writing the review itself?

Matt Zoller Seitz, RogerEbert.com, Vulture

If it were up to me, RogerEbert.com wouldn't give star ratings, because too often readers either get hung up on parsing them or don't even read the reviews because they mistakenly think the star rating covers the "gist" of the reviews. Star ratings, numeric ratings and other ranking systems are also problematic because they are pretty much useless when it comes to measuring what used to be known as a mixed review -- you know, where you think the cinematography and the lead performance are utterly brilliant but the rest of the movie is forgettable. What's that? Two-and-a-half stars? What if it's a mid-period Marlon Brando film where it's worth seeing because Brando is amazing? When Metacritic and Rotten Tomatoes come into the picture things get even more complicated. Still, I understand why people enjoy parsing numeric ratings and stars, and why critics enjoy reading other people's top ten lists even if they don't particularly enjoy having to make their own. I'm mostly against all this sort of stuff because I'm a romantic about criticism. But the fact that I keep making lists and assigning star ratings tells you that I'm either a realist or a hypocrite. I do know that I wouldn't trade those Sight & Sound lists for anything.

Keith Phipps, The Dissolve

I'm now the veteran of two different grading systems and, honestly, I prefer it. It gives readers a frame for the review and it removes the option to deliver a lot of equivocating thoughts that don't really arrive at a conclusion. (I should clarify that I'm not talking about any writers that work for me so much as my own worst tendencies as a writer.) What bothers me is when readers put too much stock in them. To me, they're meant to be looked at, absorbed, but then superseded by the words of the reviews themselves, which will always have a lot more to say than any ratings system ever could.

Glenn Kenny, RogerEbert.com, Some Came Running

For the first two years I was the film critic at Premiere we did not have any kind of rating. Oh, what a paradise it seemed. It was. Then we had a new editor, and I was terrified that I was gonna get fired because I was a drunk irresponsible blowhard asshole who never got to the office even vaguely on time, so when he asked, "Is there any reason we don't run star ratings in the reviews section?" I said, "Oh no, no reason at all, absolutely not, would you like star ratings in the review section if you like and I can figure out how to do it I can make it retroactive do you want a base four or base five?" And so we had star ratings in the reviews section. And then that editor was relieved of his duties, less than a year later, and the next editor, when I asked "Can we go back to not having star ratings in the review section?" (for some reason I had gotten a little of my gumption back), thought about it and said, "You know, once you start that sort of thing it's really hard to go back." Using star ratings makes your work more blurb-able and recycle-able for advertising and promotion people, and they like that, and they don't like when you try to take that away from them. So, you know, sure, I'd love to review movies and not have to star-rate them. I'd also love to do a lot of other things that I don't do. And I don't begrudge the sites that use star ratings one bit. In the current clotted competitive atmosphere etc etc etc. Actually it's kind of fun to make the differentiations involved in awarding stars. Doing it for MSN was a stitch because they used a five-star system which allowed for some droll (at least to me, in the privacy of my own assessment process) distinction-juggling. And as a long-time Christgauite I've always been rather fond of the letter-grading system, although I think that works best with his short-form style.

It occurs to me now that I could have saved a lot of time by just answering "Whatever." 

Joanna Langfield, The Movie Minute

I have always bristled when editors and program directors encouraged me to "grade" my reviews. It was hard enough when I had to consolidate my thoughts from a rambling 4 hour talk show into the admittedly far more commercially viable "Movie Minute" (although it has been an interesting challenge, honing and self-editing, aching for the one right word instead of an easier loose babble)! But even those of us who refuse, or have been lucky not to have been pushed, to assigning stars have to deal with juicy or squishy tomatoes, if we link to Rotten Tomatoes. And back in the day when Variety editors would call us, asking if our reviews were "positive", "mixed" or "negative", we were eventually told to stop using the "mixed" option so often. "But what if my feelings really ARE mixed?" I asked because, honestly, I almost always see high and lowlights in the pictures we review. The answer? Essentially, it was a strong request to get over myself and pick a side. Like, you know, Ebert and Siskel: thumbs up or thumbs down. 

Scott Renshaw, Salt Lake City Weekly

I've used a numerical/star rating scale of some kind for as long as I've been writing criticism, and I've hated it almost as long. Deep in my heart, I recoil at the idea that a reader won't look through the entirety of my nuanced argument, and will instead possibly rely on the shorthand, or maybe even proceed to wonder why the 2-1/2 star review I wrote this week sounds like a slight recommendation, while the one I wrote a month ago sounded like a slight dismissal. While it's been the tradition of my outlet to include a star rating with film reviews, no such metric accompanies reviews of local theater productions, or books, so I suspect if I pushed hard enough to kill the star rating, I'd get my way. Yet on some level, I understand that I can serve different readers in different ways. We can't pretend criticism doesn't also become a consumer guide, and to the extent that we can serve readers efficiently in that area, star ratings have a certain value. And I also like to think that I've shown enough restraint with both extreme ends of my rating scale that readers might really want to dig into a rare 4-star rave or zero-star evisceration. So the bottom line is: I wouldn't miss the star rating if it were gone, but I'm not prepared to invest all of my professional capital in its eradication. There are bigger battles to fight.

Stephen Whitty, The Star-Ledger, Newhouse

Stars, letters, hats, apples, clapping men -- I've seen and hate the lot of them. You don't see them on book reviews, or play reviews, or classical-musical reviews -- just movie reviews and restaurant write-ups. Why? Because I think both types of writing are generally, sadly seen by most outlets as consumer guides, not as real criticism.

We know better, of course.

I use stars because I have to, but they're problematic for a variety of reasons. I can agonize over the difference between two-and-a-half or three; wonder if I'm rewarding movies for succeeding at limited goals, and punishing movies only for daring too much.

Over the years, I've developed my own internal scale. Four stars? Buy this on Blu Ray when you can. Three stars? Go see it in a theater. Two and a half? Worth a look, as a rent. At one and a half stars and below, a movie has to have done something to ACTIVELY offend me. 

James Poniewozik, Time

Hate, and I wrote about this at length forever ago -- ("What other writer is regularly expected to append his work with a shorthand tag to help people avoid reading it altogether?"). But since I probably hate it for the same reasons many other critics here do, I'll say where I DO find stars, grades, and point ratings useful: in aggregations of recommendations. If I'm looking for somewhere to eat in a strange city, I freely admit I'll go on Yelp and look at the star ratings. But if I then want to go drop some serious bucks on a restaurant, I might then read a lengthy critic's review -- and then I don't care whether he/she gives the restaurant an A or an 8.5 or 4 1/2 sporks, I want to read the prose. Basically, the more I value your individual opinion, perceptiveness, and taste, the less I care about the grade you give something.

Robert Levin, amNewYork

In an ideal world, readers would read film criticism because they loved the art, they appreciated every last flourish and nuanced argument and admired the ways a well-written review can see a film in a whole new way. We all know that's not the mindset of 95% of the criticism-consuming population, especially in the Rotten Tomatoes era. Once you've reached that understanding, you're faced with a choice: Become one of the lucky few who works for a publication that is high-minded enough to live without star ratings, or accept the reality and embrace the fact that, for better or worse, readers are drawn in by the stars. I look at ratings as a challenge, a motivator to write well enough that my readers are as interested in what I have to say as the fairly arbitrary indicator attached to it. I probably give the specific ratings too much thought in the grand scheme of things, but as I've continued to do this I've become more flexible when it comes to last-minute shifts or changes; I see them as wholly malleable and forever subject to reconsideration and I've stopped perseverating on the difference between two-and-a-half and three stars etc.

Ed Gonzalez, Slant Magazine

I loathe rating movies on a scale about as much as I do comments sections and placating the spoiler police, and I wish my job never entailed having to answer questions such as "Does this piece sound like a two-star or two-and-a-half-star review?" or "Should this be a rotten or ripe tomato?" I would prefer to live in an alternate universe where Slant Magazine didn't rate movies using a star system, where people came to our reviews and actually engaged with our criticism, where we didn't have to deal with individuals calling for our heads because we gave "12 Years a Slave" the same star rating we gave "Neighbors." But as has been drilled into my head over and over by fellow shot-runners on the site, as a business practice, it's a necessary evil, and its practicality has at least been confirmed over the years by individuals who claim to come to the site prior to seeing a movie in order to take note of our star rating before then returning to read the actual review after seeing the movie.

Richard Brody, the New Yorker

It's a mistake to assume that words can be translated into numbers or numbers translated into ideas, as seen in the trivial ranking of critics by supposed consensus-value. Criticism is writing, the creation of an experience arising from the experience of watching movies -- but, still, there's an element of hierarchy to that experience, an assertion of merits and preferences, of tastes and distastes that is essential to criticism and, for that matter, to the best criticism. A review doesn't need righteous austerity or clinical detachment to be substantial and worthwhile any more than a movie does; the problem with numbers isn't their opinionizing but their one-dimensionality. I recently wrote a pretty negative blog post about a movie and was delighted to hear a few days later from a colleague that it made her want to see the movie; that was my intention, and I doubt that a translation into a number or a letter grade would have conveyed it.

Peter Labuza, The Cinephiliacs, Masters of Cinema

Dan Sallitt said to me that even if you don't do ratings or rankings, you probably have some internal mindset that is doing them for you anyways. I keep an internal record for my own tastes, and include it in a review if I'm asked to. I put as little thought into as possible, which is what I am now doing for this question.

Kevin Lee, Fandor, Sight & Sound

Like so many things in life, there are rules, and then there's what you do with the rules. Ratings and letter grades are inevitable, because people desire bullet points and short cuts and one-stop-shops to get through life with minimal complication. I suspect that if a clinical study were conducted to see just how many people actually read beyond the letter grade to the actual review, the findings would depress a lot of critics (and make their editors wonder if that space were better filled with ads -- then again a lot of reviews basically amount to ads). In the face of these realities, it becomes imperative to fuck things up a little. Rosenbaum was the first critic to make me realize that a rating could be used as a rhetorical device, and a provocation to deeper engagement. Back when he gave two stars to "Fargo" in the midst of the film's unanimous acclaim, my first thought was "Is the son of a bitch out of his mind?" Of course, I had no choice but to read the review. I came out of that review with a totally new perspective on the Coen brothers that stays with me to this day. So rather than reject a rating or a grade system, a critic can be resourceful about how to use them to engage a reader. In some ways doing it well requires as much intelligence as the crafting of a memorable sentence.

Jordan Hoffman, ScreenCrush, NY Daily News

I don't mind doing them too much. It is a reasonable shorthand. As a freelancer with many outlets, however, I deal with this nice bit of comedy: one of my outlets does letter grades, another uses a 1 out of 10 scale and others use stars. And one outlet allows for half-stars, while the other does not. What's annoying is the apples-and-oranges effect. A very ambitious movie with marvelous scenes but a flubbed ending can end up with 3/5 stars, same as a wafer thin comedy that had some quality zings but not much else. One outlet that I work for on a once-in-a-blue-moon basis, Badass Digest, uses no grading whatsoever. I must confess that when I file for them it's a bit of a fresh breeze.

Luke Y. Thompson, Topless Robot

I LOATHE having to give a rating, especially if it has to be as limited as a scale of 1-4 stars. It gives people an excuse to ignore my actual argument, and use it as a metric to invalidate any given review by saying, for example, that I gave "Sharknado" a higher star rating than "Schindler's List," when you really cannot compare those movies to each other in any meaningful way -- other than my arbitrary number ratings of them that I am theoretically required to give.

Readers LOVE ratings, though. They also love ranked lists. So even though I don't currently give movie rankings, I probably should. And if I must, I'll go with letter grades as being among the best, though I don't entirely agree with Rotten Tomatoes that anything below a B- is "not recommended" (i.e. "Rotten"). Relatively little time goes into conjuring that rank when I have to give it - beyond the aforementioned Rotten Tomatoes B- divide distinction.

Alonso Duralde, The Wrap, What the Flick?!

I would happily never have to do give a number/letter grade score, mainly because I think it makes it easy for people to skip reading the review and just grasp the yea/nay-ness of it all. But since I currently have to (not for my own outlet, but for other places where my my reviews run), it's definitely something I think about only after the writing is finished.

Neil Young, Hollywood Reporter; Tribune

I fully accept the absurdity and arbitrariness of ratings, stars and scales, and it can't be a coincidence that none of my favorite critics (Danny Peary, Graham Greene, Pauline Kael, James Agee, Vern) ever went anywhere near them. However, I'm not those folks, nor was meant to be, and I've deployed stars/ratings/scales of one form or another since I started watching films on any kind of methodical basis, i.e. when I kept tallies of my horror-film viewings on TV from the age of 9, in 1980. Over the years this evolved, through various fits and starts, into Jigsaw Lounge's long-notorious out-of-28 scale (maintained in tandem with a universally-graspable out-of-ten-but-please-no-zero assessment) which makes total logical sense to me and baffles the bejesus out of most everyone else.

It's a gut thing: only rarely will my final assessment differ significantly from the "grade" I scribble down in my notepad (often in the preliminary form "6^/6" -- i.e. 17/16 out of 28) towards the end of a film. I don't spend much time on this aspect at all, especially in comparison with how long I spend note-taking, note-typing-up and review-writing; but I do find it a very handy way of getting to mental grips with the daunting surfeit of pictures we tireless critics are expected to cope with month after month, year after year.

Ratings are bunk! Long live ratings!

Features
  • |

More: Criticwire Survey

2 Comments

  • Sean | May 12, 2014 10:53 AMReply

    Because I tend to watch movies before reading about them (to avoid coloring my interpretation too much), I generally find the ratings pretty unhelpful and often contradicted by the review itself. I think a binary type system would be more useful - less of a good/bad but rather a watch/don't watch. Richard Brody wrote a fairly negative review of Under the Skin that also conveyed that it was essentially a must watch film, something a star could not do.

    In college, I was the film editor for the campus newspaper and my first order of business was dropping the letter grade. The head editor demanded something, so I instituted an intentionally absurd scheme of animal ratings (e.g. a movie could get a koala, a crocodile, or a whale). I was quickly forced back to letter grades, but I resented it.

  • zé | May 12, 2014 10:34 AMReply

    Ratings are very helpful, because "normal people" don't have time for watching every movie there is, so we have to choose somehow. Usually my choices are based on the ratings, because I don't like reading reviews before watching the films. Those who'd rather experience a film without reading anything about it beforehand would have a hard time without critics' ratings.

Email Updates