CalSPEED Still Going Strong!

CalSPEED, our crowdsourcing app, continues to provide a steady stream of data for the California Broadband Interactive Map. Only halfway through April, we have over 140 CalSPEED tests, which is one par with our quarterly average to date.

Spring 2016 Field Testing Has Started

photo by Steven Crews

Monday Evening Sushi Buffet - T-Mobile's "Binge On" Reconsidered

As more people have dug into T-Mobile's Binge On service, it has become clear that it's not as great as it sounds. It starts to look like the Monday evening sushi buffet -- the fish market is closed on Sundays, and all that leftover fish from Saturday night must be sold! Which would you prefer, a $50 steak dinner, or a $10 all-you-can-eat sushi buffet? Depends on your budget and what you're willing to put up with.

My earlier analysis of where you might be likely to stream high definition video using T-Mobile's Binge On service did not take into account the fact that streaming HD video with Binge On may be impossible, because T-Mobile throttles the speed down to 1.5 Mbps. The Electronic Frontier Foundation did a test using Binge On and determined that their video optimization is "just throttling," and that T-Mobile applies the speed throttling "indiscriminately to all video."

Does that make the streaming video coverage map in my earlier post wrong? No, but the only way you will likely be able to receive HD quality streaming video is by disabling the Binge On feature.

Another way of looking at Binge On, suggested to me by a colleague: T-Mobile is simply "incentivizing lower quality video usage … lower load on the network [coupled] with kinda an acceptable level of quality."

Coachella Live: Will My Phone Work?

For two weekends next month (April), a bunch of bands will be playing Coachella at the Empire Polo Club in Indio, California. In case you or someone you know is going, check out the mobile broadband maps to see what level of service you might expect in that area.

T-Mobile, which tends to have weaker rural coverage than either AT&T Mobility or Verizon Wireless, nevertheless appears to have strong coverage around the Empire Polo grounds on 51st and Monroe. Maybe that's not surprising given that T-Mobile's target demographic is the youngest among the four mobile providers. For Sprint, our model suggests there is no service in the area of the concert venue, but we encourage concert-goers who have Sprint to download CalSPEED (free download from iTunes or Google Play) and run some tests for us, because those test results will help us improve our model. The one CalSPEED test we have from crowdsourcing for Sprint in downtown Coachella shows no service, but a field test location near Harrison and Airport Boulevard shows over 7 Mbps down and over 4 Mbps up.


Here are images showing the interpolated mean minus two standard deviation downstream speeds for each provider for the area around Coachella. Concert venue is indicated by a star.

 T-Mobile: 3.0-6.0 Mbps downstream

Verizon Wireless: 3.0-6.0 Mbps downstream

AT&T Mobility: 1.5-3.0 Mbps downstream

Sprint: No service

iPhone vs. Android - Which Is Faster?

Earlier this year, we devoted a week of testing to compare the iPhone 6 with the Samsung Galaxy S6, and the Samsung Galaxy Tab 4. You might expect the iPhone to be the fastest for all providers, but it was true only for T-Mobile downstream. Below are graphs comparing the mean downstream (sending) and upstream (receiving) speeds for each (in blue), as well as the average standard deviation (in red).


We found that the GPS readings for the iPhone were slightly off. We know this, because all three devices for the four providers were tested in a stationary location each time. The plot above shows how some of the iPhone locations appear slightly off from the others.

 

 

Bicoastal Speeds Are Converging: Sneak Peek At Upcoming Novarum Report

I've touted the benefits of CalSPEED's use of servers on both coasts of the U.S., and so you may be asking, "How's that working out for you?" It's showing an interesting trend: convergence. In the upcoming Novarum report on the Spring 2015 mobile field test results, Ken Biba shows how speeds to west and east coasts are converging. In the graph above, we see the difference in speeds for any location where user tests to both the west coast server and east coast server. For Sprint, the east coast speed is 10% slower, and for Verizon, it is about 20% slower.

From the Novarum report:
"In the best possible time, the physics of data transmissions adds about 80 milliseconds of additional latency to get from one coast to the other, in addition to any local wireless access latency. In the case where the  latency difference between servers on opposite coasts is zero, we speculate that traffic for both servers is peered through a geographically central location, such as Kansas. The latency difference between East and West servers has decreased over time - converging on 80 milliseconds."

Why the FCC Should Adopt CalSPEED, Part 2

In Part 1, I briefly went through three mobile data-related questions that underlie the FCC’s Big Policy Question regarding Advanced Communications Services as defined in section 706 of the Telecommunications Act of 1996.

In Part 2, I’ll show how the mobile data the FCC is currently using falls short in answering the Big Policy Question and why the FCC should adopt the CPUC’s mobile testing methodology.

CPUC Mobile Field Test Data

The CPUC conducts a statewide field test of the four major mobile providers twice a year across 1986 locations covering 52% rural, 37% urban, and 11% tribal. The test relies on both a west coast server and an east coast server in order to mimic the user experience for both cached and non-cached content. The test uses both an Android smartphone and a tablet in order to get a profile of more than one device and does not filter any of its results - throughput, coverage, latency or other network metric.

PRO

• Tests at same location twice a year
• Point data results are interpolated to create “heat maps” of speed, adjusted speed, latency, VoIP, streaming video, and video conferencing.
•  Test works where there’s no network. “No service” is recorded and the test result is uploaded to the CPUC’s database the next time the test is run in a connected environment.
• Results from crowdsource version of test (CalSPEED) used for validation
• Code is open source, available to anyone to modify for their own purposes (it’s being used by Virginia Tech—search for “Data Cardinal” on Google Play)

CON

• The FCC references results from the CPUC's mobile field testing program, but test results from the semi-annual mobile field test are available only for California.

(Please visit the CPUC's mobile broadband testing page to download the latest reports, data, and overview of the CalSPEED mobile testing application back-end design)



Form 477

The FCC relies on Form 477 to collect “information about broadband connections to end-user locations, wired and wireless local telephone services, and interconnected Voice over Internet Protocol (VoIP) services in the 50 states, the District of Columbia, and the Territories and possessions.” The FCC uses 477 data to map broadband infrastructure deployment and estimate population coverage. Mobile operators are asked to provide their minimum advertised speeds.

PRO

• Form 477 has been used for a number of years to collect broadband subscriber data from providers. The addition of deployment data to the submission process following the end of the NTIA’s State Broadband Initiative means that the FCC can get both data sets directly from providers.

CON

• Self-reported data
• Who advertises minimum speeds? The only reliable minimum speed for mobile is zero, which you get when you leave a network coverage area.
• Minimum advertised speed isn’t a reliable proxy for actual speed. The FCC says: “The relationship between advertised and actual speed is more complex for mobile services because the mobile providers report their minimum advertised speed and each mobile provider advertises the minimum speed at various points of their actual speed distribution.” (2016 Broadband Progress Report, footnote 246).

Rather than dive further into that topic, let’s compare the FCC’s estimate of population coverage for 10 Megabits per second down and 1 Megabit per second up (10/1) with the CPUC’s estimate of coverage for a slightly lower threshold. The CPUC’s mobile mean throughput speeds (upstream and downstream) are adjusted by two standard deviations and then used to create a “heat map” indicating likely coverage between test locations.

• FCC: 53% “of Americans don’t have access to [a] mobile service provider with a LTE technology service with a minimum advertised speed of 10 Mbps/1 Mbps.” (paragraph 83, Table 4). By extension, 47% of Americans should have access to LTE at a minimum of 10/1.
• CPUC: Only 16% of California’s population has access to speeds of at least 6 Mbps down and 1.5 Mbps or greater using the adjusted mean throughput.

How does this data help answer the Big Policy Question? The FCC recently revised the definition of advanced telecommunications services to the level where “consumers have access to actual download (i.e., to the customer) speeds of at least 25 Mbps and actual upload (i.e., from the customer) speeds of at least 3 Mbps (25 Mbps/3 Mbps).” (paragraph 26). Note the use of the word, “actual,” and not “advertised” or “minimum advertised.” In this regard, the CPUC data is more suitable than 477 data, because the CPUC data is more granular and is based on actual tests.

Mosaik
Mosaik is a private company that advertises having the “largest mobile network coverage database in the world." The FCC's map page shows a nationwide map of mobile coverage based on data from Mosaik and the 2010 Census.

PRO

• Mosaik provide a one-stop shop for mobile coverage data.

CON

• How accurate is the data? According to the FCC, Mosaik data may overstate coverage: "Coverage calculations based on Mosaik data, while useful for measuring developments in mobile coverage, have certain limitations that likely result in an overstatement of the extent of mobile coverage."
• What service level does the coverage data represent?

As I explained in a previous blog post, you can’t show coverage without specifying service level. What level of service does the Mosaik coverage map indicate? It’s not clear. Moreover, the image below suggests that most of the state is “covered” by at least one major provider.

The Mosaik data is binary: green (coverage), and white (no coverage).
In contrast, here is the CPUC’s mobile broadband coverage map image of California based on a service level of at least 6 Megabits per second down and 1.5 Megabits per second up using interpolation of mobile field test results of mean minus two standard deviations.


Rather than showing a binary “coverage/no coverage” image, the CPUC’s map (lower left) shows areas where an end user is likely (e.g. mean minus two standard deviations – see explanation here of why we do this) to get 6 Megabits per second down and 1.5 Megabits per second or greater (green). Yellow indicates areas where a user is likely to get speeds below that threshold, but higher than combined 768 Kilobits per second down and 200 Kilobits per second up (768/200). Finally, red indicates areas where not even 768/200 is likely.
The CPUC’s data provides more granularity than Mosaik’s, and it shows different service levels, depending on the geography. Furthermore, as I mentioned earlier, the CPUC data lets us map different types of service levels such as VoIP, streaming video, and video conferencing.

FCC's mobile testing app (mMBA)

In 2013, the FCC released a mobile testing app on Android and later followed up with a version for the iPhone. The purpose of the app is to crowdsource mobile performance across the United States. The FCC states that its speed test app “accurately measures mobile broadband (cellular) and WiFi network performance and delivers consumers an in-depth, real-time view of key metrics related to their mobile broadband experience.”

PRO

• Provides the FCC with real-world mobile performance data

• Crowdsourcing lets the general public become more informed stakeholders in broadband policy (the same reason why the CPUC also offers its own crowdsource app (CalSPEED) on the iPhone and Android.

CON

• Where is the data from the mMBA tests? One of the few snippets of mMBA data appeared in the 2016 Broadband Progress Report: "On a national level, mMBA reports median LTE download speeds for the first two quarters of 2015 as 11.6, 7.5, 5, and 13.6 megabits per second for Verizon, AT&T, Sprint and T-Mobile, respectively." 
• Crowdsourced data provides input only where the crowds are -- namely, mostly urban locations. This is why the CPUC conducts a separate, semi-annual mobile field test.

CONCLUSION

Ideally, data should line up perfectly with policies. It can be both costly and difficult to do make that happen, but it’s good to be aware of the mismatch and have an idea of what data is needed to close the gap.

In an ideal world, every American would have the FCC’s app installed on his or her smartphone, and the app would provide petabytes of actual mobile broadband data to the FCC and the general public. Unfortunately, that doesn’t seem to be happening. In place of that, there may be a way to take the methodological and analytical approach we’ve created at the CPUC and apply it on a larger scale, nationwide drive test of geographically representative locations.

-Special thanks to Karen Eckersley and Ken Biba for their valuable feedback.