The 5-Star Nonreview
Lots of developers use the App Store review section to interact with customers. They might, for instance, announce future upgrades or offer a rebuttal to user gripes. And many use the rating system to pad their app's overall score, even though they're not reviewing it.
When Chris Meyer posted a rebuttal to a critical, 2-star review of his AirPhones audio-streaming app, he gave his own program a 5-star rating. "Sure, I gave myself 5 stars, but that was in knowing that the average of mine and the other 2 star [review] would be reasonable," Meyer told PC World via e-mail. "Also, I only have one opportunity to tout my app, so if there are dozens of comments, it won't matter much."
Meyer added that AirPhones has received several "harsh and unwarranted" 1-star ratings, and that he suspects that some of those reviews were "intended to hurt the product." But despite the imperfections of the App Store review process, Meyer sees it as a "wonderful example of a democracy at work."
In recent months, news reports have stated that some iPhone developers are offering to pay App Store users for positive reviews. In November 2008, for instance, Wired ran a screen shot of an ad on Amazon's Mechanical Turk job site that promised $4 to any App Store user who would post a 5-star review of the SantaLive application. (The ad was later removed.)
To find out if such a practice is common, I searched 560 review-related postings on Mechanical Turk. I encountered only one App Store developer offering to pay for reviews. The fee: 30 cents for any review, good or bad. I submitted two reviews of the product, one 5-star (very positive) and the other 1-star (very negative), and received payment for each.
I also searched Craigslist job posts for 25 major U.S. cities, but I didn't see a single instance of an App Store developer offering to pay for reviews. Though anecdotal, my informal survey suggests that reviewing iPhone apps for pay isn't a huge industry--but that doesn't address how often family/friend/employee reviews distort ratings.
Still Too Easy to Cheat
Apple has taken steps to limit the number of questionable App Store reviews. In September 2008 it changed its review policy: Now, only users who download a particular app can review it. And earlier this year Apple deleted comments from customers who hadn't actually used the programs in question. (In an unrelated but also helpful move, Apple recently began sorting user reviews by version as well, so that at least you have some context when you're reading a review.)
While those measures have eliminated many suspicious reviews, it hasn't prevented users from reviewing apps they haven't tried--users need only purchase and download them. Using the Windows version of iTunes, I downloaded two App Store games, Tap Tap Revenge 2 and Wrangle Lite, to a laptop PC. And although I never actually installed the games on an iPhone or iPod Touch, I was able to post reviews of each title.
For App Store customers, a healthy dose of skepticism is a good thing. "I haven't put much stock in the reviews from the App Store," wrote blogger Nick Santilli in an e-mail interview with PC World. Santilli, who has blogged about questionable App Store reviews for TheAppleBlog, would rather hear about useful apps "from the Twitter cloud," or by speaking directly with developers or fellow bloggers. "These methods seem much more reliable than the mass of random users in the App Store review system," he wrote.
Finding Fake Reviews
How can you determine whether an App Store review is credible? One way is to seek out and read other reviews the same customer has posted; for instance, if someone has posted only 5-star reviews for apps from a certain developer, that's a red flag. Unfortunately, this investigative work isn't possible if you browse the App Store via an iPhone or iPod Touch, but you can do it in the iTunes desktop interface. Apple should address this shortcoming, since downloading and installing apps over the air is much easier than doing so via a PC. "The majority of people are downloading [apps] through the App Store interface on the device itself, just because of the seamless behavior," says ABI Research senior analyst Jeff Orr, who recently studied the spending habits of smart-phone owners.
Despite the system's flaws, however, App Store reviews still provide valuable insights. "Reading reviews on some of the apps I've used personally, a random sampling seemed spot on and generally fair," wrote Santilli. "People are so used to Apple getting things right the first time...so when every angle hasn't been considered right out of the gate, the frustration is made known from the rooftops."