Today I'm interviewing Marc Doyle, co-founder of review collector Metacritic. I'm known for griping about Metacritic's approach to game score aggregation, so it promises to be interesting. The CBS Interactive-owned company's single "unifying" game percentile tends to do anything but unify. Well, in my line of work anyway, where if you tick off a publisher with a lower-than-average game score, you're liable to fall off their mailing list.
But that's technically not Metacritic's fault, and Marc's always seemed like an affable guy willing to listen to criticism in emails. I dial the conference number.
"Hi Marc, am I hearing a British accent?" I ask after we're connected. I'm running the interview over Skype. For some reason Marc sounds like Colin Firth.
"No, no, born and raised in sunny California," says Marc, suddenly sounding more like Seth Rogen. Thanks a lot, Skype.
"So I don't think we've ever talked before," I say.
"No, I don't think we've even met," answers Marc.
We banter a bit about that and a few emails exchanged over the years. About where we might've met. About whether he goes to trade shows like E3. He says he used to, that at one point he hit E3 seven years in a row. But not anymore. He says it's because Metacritic's "more about the post-release process and not the pre-release hype that goes on with E3." We commiserate about the latter before I dive in.
"What was the thinking behind adding career scores for developers?" I ask. I'm talking about a feature Metacritic recently added--then yanked--that tallied the scores of games by noted developers and assigned each developer an average overall score. The catch: Your score was affected by anything with your name on it, whether your role was "creative director" or something tangential, like "executive producer."
"We re-launched Metacritic in August of last year, and the thought was, we wanted to be much more interactive, to allow users to discover more products," says Marc. "So for example, if you were on a movie page, we'd allow you to close on a director or writer or star and see more movies by those people. It's as simple as that, really."
But this is Metacritic we're talking about, not IMDB, so there's that not quite "as simple as that" Metacritic angle.
"With Metacritic, not only can we show that person's entire inventory, we can show what those individual movies scored on Metacritic," continues Doyle. "It's the same on the game side. If you're looking at a game that's made by Atari and you click on ‘Atari', you can see their entire inventory and the individual meta-scores of their games."
One step further, of course, and you've moved on from publishers or developers as collectives to the individuals that make up those entities. Think of it like school, only you're graded not only on the tests you've taken, but for every time you're caught standing in a room with other test-takers.
"The idea of adding individuals, say the people who make the games from the level designers to the directors, is something that came from GameFAQs [another CBS Interactive affiliate]," says Doyle. "We just wanted people to be able to say ‘Hey, I loved this game, who was behind this game? What else has this person done?'"
Which led Metacritic to throw together a "simple average" score of the games with which individuals had been involved--"involved" being the operative term. Lots of game design celebs lend their names (and money) to games they've had little or nothing to do with, creatively.
"I think the problem was that people started to say ‘Wow, this is a rating of me and my work personally', and that's not what it was intended to be," says Doyle, adding that it "was never meant to be an evaluation of a person or a career."
Doyle says the real problem isn't with the idea itself, but that the backend data wasn't granular enough: "We realized that the GameFAQs database wasn't as comprehensive as it needed to be, that we had only this small subset of each person's career, and that's why we removed the scoring system."
Which implies that had the data been there, they would've left the system up and running--a point I'm still not sold on.
"Okay, but isn't the real problem that you take a guy like Shigeru Miyamoto, credited on all sorts of Nintendo games he's had little or nothing to do with creatively, and yet those games count toward his average?" I ask. The mathematical problems with this, assuming you want a scoring system that reflects someone's creative output (to say nothing of team contribution weighting) should be obvious.
"None of these things are weighted," admits Doyle. "This was just a simple average of the individual games, so whether someone was involved just a small amount or they ran the whole show, the rating counted the same. So it probably should have been worded differently than a person's 'career score', but regardless, it didn't seem appropriate, so we decided to yank it."
But not for good. Doyle reiterates the point about the data being incomplete and calls that the "fundamental reason" they stepped back--not any media or public pressure. "Hopefully we're going to be able to create a database, which is reliable," he says. "The whole purpose of the feature was so you could further explore what people have worked on."
The latter's a noble enough idea. You can "further explore" what directors, actors, and others have worked on at IMDB. MobyGames attempts to do this for the games industry, but a lot of the information's out of date. The last entry for wargames developer John Tiller is Panzer Campaigns VI (2002). He's worked on dozens of games since. No one's attempted to build a game industry database that's truly comprehensive.
"What if you just curated it?" I ask. I dislike score aggregation, but if I had to pull together someone's project scores, I'd at least try to account for their level of involvement. "Couldn't you just curate and weight it somehow?"
"I'm sure you could go that way," says Doyle. "But again, even if you're just a lower level person, just to be able to see they worked on this or that game, you'd find that interesting, right?"
"Sure," I reply, realizing we're not connecting on the "level of involvement" thing. And again, that Metacritic might eventually offer an intuitive means to track someone's career game by game (career scores notwithstanding) sounds like a great idea. But much as I want to pursue the point, I'm on the clock, and it's time to push forward--to talk about my issues with the scoring system itself.
Next: How Metacritic works, and why...