I tested four phones on AT&T service in two cities over three days. I drove around San Francisco and Los Angeles comparing the performance of the iPhone 4 and the iPhone 3GS against the performance of the RIM BlackBerry Bold 9000 and the Pantech Impact in voice calls placed at roughly the same time from areas where coverage from the AT&T network is less than optimal.
What I found was surprising. Calls on the iPhone 4 and iPhone 3GS failed to connect or dropped in midcall far more often than did calls on the other two phones, and the iPhone calls that connected successfully sounded marginally worse than calls placed with the BlackBerry and Pantech phones.
My testing is not meant to be scientific or definitive, but the results raise real questions about the world’s favorite smartphone: Is the iPhone a great personal computing device but a bad phone? Please click the chart below to see average call quality scores (on a 1-5 scale) for all test calls made in San Francisco and Los Angeles.
Testing Methodology
I made test calls from seven medium- to low-signal locations in San Francisco; then I validated my results with a second round of testing at the same locations two days later. Finally, I performed similar tests from five locations in Los Angeles. I tested from various locations, including a parking garage, a forest, a train station, a library basement, and a moving bus. I noted each dropped or failed call, and scored each successful call that I made using the five-point Mean Opinion Score (MOS), a scale developed by Bell Labs to quantify call quality. (Please see the “Mean Opinion Scoring Guide”–the rightmost column in the accompanying chart–for definitions of the possible call quality scores, 1 through 5.)
At locations where my first call on a given phone dropped, I noted the drop and then placed an additional call, applying a MOS number to the second call if it completed successfully. Obviously, I couldn’t give a voice quality score to a phone if it failed in both attempts to complete a call at a certain location.
The Dropped-Call Champions
The data point that sticks out in my results is the number of dropped or failed calls placed by iPhones during my tests, especially in the San Francisco tests. In total, the iPhone 4 dumped or couldn’t connect in half of test calls — 14 out of 28 — in low-signal areas. The iPhone 3GS did even worse, connecting only 12 calls in 30 attempts, for a success rate of 40 percent.
Meanwhile the iPhones’ competition in our tests proved far more reliable in dealing with less-than-ideal signal strength. The least expensive phone in our tests–the Pantech Impact–had zero failed or dropped calls in the 19 test calls I placed on it, despite poor cell conditions. The BlackBerry Bold 9000 was also far more reliable than the iPhones, connecting on 18 out of 21 test calls in Los Angeles and San Francisco.
Call Quality
I also evaluated the voice quality on calls that connected successfully. In my conversations with a colleague in the PC World office, I listened for things like drop-outs, static, thin or fading voice, delay, and garble, and I gave each call a MOS score based on the cumulative disruption caused by these imperfections over the course of the call.
Here, the results were more favorable to the iPhone, and much closer across the four phones I tested. All four phones earned average MOS score of between 3 and 4 (that is, between “annoying” and “fair”), a range that is said to be typical of calls placed on cell phones.
On average, the iPhone 3GS scored the worst of the four phones in call quality. The 12 calls (out of 30 total) that the 3GS managed to connect and hold received an average MOS score of 3.16 (out of a possible 5). The iPhone 4, with its improved antenna (provided you hold it just so), fared a little better than its predecessor did in my call quality tests. The 14 (out of 28) successful test calls placed on the iPhone 4 averaged a mark of 3.27, also between “annoying” and “fair” on the MOS scale, though again, like the iPhone 3GS’s score, closer to “annoying.”
The BlackBerry Bold 9000 produced the highest-quality voice calls overall in my tests. Calls made with the Bold 9000 had an average MOS score of 3.77 across 18 test calls, putting it near the upper end of the range between “annoying” and “fair.” The Pantech Impact also outperformed the iPhones in call quality, with an average MOS score of 3.61 for its 19 test calls.
Conclusions
The iPhone’s poor performance in my tests could have been caused by myriad things, from the tuning of the iPhone antenna to the amount of network resources AT&T allocates to voice calls placed by iPhones specifically.
Neither AT&T nor Apple offered any sort of explanation. In response to a request for comment, an AT&T spokesperson said merely: “We recommend you reach out to the device manufacturers.” Apple did not return our calls requesting comment.
The iPhone is undeniably a superb device for browsing the Web, playing games, and watching videos. Further, the iPhone 4 clearly improved on its predecessor in microphone quality, speaker quality, and noise cancellation effectiveness, all of which help make voice calls sound great–when cell service is strong.
But at least for now, weak cell-signal zones are a fact of life for just about everybody, whether you encounter them on the road, in your office, or in your living room. Cell signals also tend to degrade when making their way through the walls of our homes, necessitating an amplifier–or in AT&T’s case, a Microcell–that, more often than not, you have to pay for. And wireless operators usually invest in additional cell towers and in radio improvement only when increased traffic levels (which translate into average revenue) are likely to be high enough to justify the capital expense.
While we don’t have the resources to do our testing nationwide, the data I collected in San Francisco and Los Angeles, combined with the anecdotal evidence, is enough to indicate that iPhones may well have more difficulty than other phones placing calls in areas of less-than-optimal wireless service.
If you’re considering buying an iPhone, and phone calls are important to you, it’s wise to take the time to test the phone for as long as laws in your state allow to understand how well the device works in the low-signal areas you frequent.