Fall/Winter Testing Over 1/3 Complete

In the two weeks we've been testing, over 1/3 of the locations have been covered. Due to the late start, our team prioritized snow-affected areas with the thought that it might be harder to test there after we resume in January. Here are a few photos:

Fall/Winter Testing finishing 2nd Week

We are finishing up the second week of mobile testing before winter break. This time around, we have upgraded the smartphone to the Samsung Galaxy S6 from the S4. The tablets are still the same as for Spring 2015 testing. It's still too early to draw conclusions about any of the providers, so in the meantime, we are working on two new areas of analysis: video streaming and conferencing capability of each provider's network, and backhaul. For the latter, we are analyzing traceroute test results from the Spring 2015 test and will be comparing them with those from this round.

The Nation's Largest Mobile Network

Who has the largest mobile network? In the recent past, red or blue maps were flashed across the television screen, and we were left having to flip a coin, because it was impossible to tell which provider had better coverage. Missing completely was any mention of service level (voice call, or Skype video call, or streaming Netflix HD, as examples) or probability of getting that service level in a particular area.

We all want to know where we will be able to use our mobile devices -- Yosemite? Mammoth? Anza Borrego? -- and we think there's a map out there that will provide the answer. But, most maps are garbage. A good map will state that the coverage shown on the map is based on the probability of getting a certain level of service at a location.

A generic map may show nearly all of California as having mobile service, but that coverage might be based solely on voice calling, not broadband. If you want to stream YouTube, the coverage will likely shrink. Moreover, if you want to stream YouTube reliably, or use real-time streaming services without interruption, the coverage may shrink even more. Think: higher capacity = less coverage, and higher probability = less coverage.

In our study of mobile broadband coverage for California, we now incorporate probability into our maps. In the past, we used to display mobile "served" coverage estimates (i.e. greater than or equal to 6 megabits per second downstream AND 1.5 megabits per second upstream, or "6/1.5") based on average speeds measured in the field. That resulted in 98% of California households receiving 6/1.5 or faster. But, as I have written in earlier posts, "average" means that half of the time you will get better than 6/1.5, and half the time, worse.

Here is our served 6/1.5 household estimate from June 2013 using average speeds:

This is why we are now adjusting the average speed with the standard deviation. The standard deviation is a measurement of variability. The higher the standard deviation, the more variability. Reducing measured speeds by the mean (average) minus two standard deviations yields a much higher likelihood (98% in a normal curve) of getting 6/1.5 or faster at a particular location, but, that adjustment shrinks the coverage map. This higher threshold lowers the estimated household coverage to 16% of California.

Here is our served 6/1.5 household estimate from December 2014 using mean minus two standard  deviations:

Remember, coverage maps are meaningless without specifying service levels and incorporating probability.

iPhone CalSPEED On Track For December Submission

We've been testing CalSPEED for iPhone, and we've been happy to share the beta version with enthusiastic members of the broadband community who've tested it out it remote areas of California. We've just upgraded the user interface as well as the back-end latency test algorithm (more accurate), which means we are on track for submitting CalSPEED to Apple in early December for review and (hopefully) approval for distribution on iTunes.

Japan: A Look Into America's Mobile Future?

For as long as I've been working in mobile telecommunications, Japan has always been the leader in mobile network development. A recent look at one of Japan's crowd sourcing apps, RBB Speed Test, shows some pretty amazing speeds for mobile data. While we might expect to see a maximum download of 60-70 megabits per second (Mbps) in California, in Japan, it's more than three times as fast. For upload, we're used to seeing between 5-10 Mbps; for the crowd source results below, the increase is not as large (17.1 Mbps).

See the table below for the top ten speeds from November 13. Number 1 was a test run on NTT DoCoMo's LTE network using a Sharp SH-01H (link to GSM Arena here) (link to Sharp's Japanese page here).

CalSPEED Shooting for 10K

Above: actual CalSPEED test locations

By all accounts, the CPUC's crowdsourced mobile testing project, CalSPEED, is an unequivocal success. Since its launch two years ago, we have had over 9,000 tests performed. We added an auto-update feature so results are shown in the interactive map within 24 hours. We have an iPhone version of the app in beta testing right now and plan to have it available to the public in December. Moreover, because streaming video is such an important feature for mobile we will be adding a video quality metric to the scorecard.

Mobile Data presented to Assembly Select Committee on the Digital Divide in Rural California

On October 27, I presented an overview of the CPUC's broadband programs to the Assembly Select Committee on the Digital Divide in Rural California. One of the key points was the apparent discrepancy in served households between wireline and mobile after we adjust mobile speeds using mean minus two standard deviations. At the 6 mbps down / 1.5 mbps up or higher threshold, we estimate 57% of rural households and 2% of urban households are either under-served or un-served by wireline. For mobile, the number of under- and un-served jumps to 85% of rural households and 84% urban households.

Should Mobile Be Included in FCC's Definition of Advanced Telecommunications Services?

 This week, the CPUC filed comments with the FCC regarding Section 706 of the 1996 Telecommunications Act (GN Docket No. 15-191, Inquiry Concerning the Deployment of Advanced Telecommunications Capability).

The comments examine some of the phenomena we have observed over the past seven rounds of statewide mobile testing and suggest that the FCC defer its decision on including mobile broadband in its definition of advanced telecommunications capability until the FCC confirms that it has reliable mobile data, and that it has mobile performance benchmarks that include critical performance metrics such as latency and packet loss.

The comments are publicly available from the FCC's web site and can be downloaded here.

Has Mobile Speed Growth Stopped in California?

This week, we posted the latest field test data from Spring 2015 on the CPUC web site. Whether due to increased traffic load, network optimization to improve other factors such as latency, or a combination of both, there appears to be a trend of slower phone speeds for the two dominant providers, Verizon Wireless and AT&T Mobility. Note that, in contrast, we saw faster speeds on the newer tablet devices. This could be due to a newer category of radio chips in the tablets. The other two providers saw minor increases in mean downstream throughput. As mentioned, we did observe improved latency, and we would expect that to improve each provider's streaming media services such as Voice over Internet Protocol (VoIP).

As if to confirm that trend, the number of locations where the average downstream speed met or exceeded 25 megabits per second and the upstream met or exceeded 3 megabits per second dropped significantly, as shown below.

In addition to the Spring 2015 field test results, the latest analysis from Novarum (CalSPEED: California Mobile Broadband, An Assessment - June 2015) is now available on the CPUC web site. This report incorporates the Fall 2014 mobile field test results, and it has some interesting conclusions, namely:

• Mobile broadband’s overall performance and quality has stopped improving and shows signs of degradation.
• Mobile broadband continues trends of wide variation across California among carriers, locations of services, the growing digital divide between urban and rural.
• Quality degradation is particularly noticeable in rural areas - in which quality metrics can be 2x worse than in urban.
• Penetration of rural LTE shows signs of stalling.
• There is substantial variation between user devices on the performance and quality of service.

Can First Responders Use Streaming Voice from Your 911 Call Location?

Did you know?
The majority (greater than two-thirds) of 9-1-1 calls in 2014 were made on cell phones.

What we did...
We compared the location of where 9-1-1 calls that went through successfully with an estimate of mobile VoIP coverage based on our Fall 2014 mobile field test results.

Below, in blue, is the estimated AT&T mobile VoIP coverage, where locations showing an estimated Mean Opinion Score (MOS) of 4 or greater are shown. Please refer to my May 26 blog post about how we calculate MOS and what it means.

Below, in red, is the estimated Verizon Wireless mobile VoIP coverage, where locations showing an estimated Mean Opinion Score (MOS) of 4 or greater are shown.

Why do this?
We wanted to get a better understanding of what parts of the state could support real-time streaming services for emergency responders who went to the location of the 9-1-1 call.

What's in the 9-1-1 data?
These images show not only mobile (cell phone) based 9-1-1 calls, but also home telephone 9-1-1 calls. For estimated mobile VoIP coverage, we looked at AT&T and Verizon. We did not consider other providers, because their coverage was significantly smaller.

Why should I care?
As FirstNet prepares to put out for bid a nation-wide interoperable mobile network for emergency first responders, it's important to understand where existing providers have coverage that supports streaming services such as VoIP, and where they don't. These two images provide a starting point for the discussion.

Mobile Field Test Maps @ FirstNet Consultation Jul. 28-30 in Sacramento

This week, we had the wonderful opportunity to present results of our mobile testing to FirstNet at the California State Consultation, arranged by the Governor's Office of Emergency Services (Cal OES) in Sacramento. The meeting brought together roughly 200 stakeholders from the first responder community at the tribal, federal, state, and local levels to learn about the First Responder Network (FirstNet) Authority's plans to roll out a nation-wide interoperable public safety mobile network using a dedicated 20 MHz in LTE Band 14.

A large part of the discussion centered around coverage of tribal and rural areas of the state, and to assist participants in their understanding of coverage today, CalOES used a series of poster-sized maps created by the CPUC and CSU Chico's Geographic Information Center. Pictured above is an image of the federally recognized tribal reservations and Rancherias with Verizon and AT&T estimated OTT MOS >= 4 coverage. Other poster topics included fire threat areas, earthquake zones, state parks, 9-1-1 call locations, population density, Yosemite Rim Fire, and FirstNet proposed initial coverage objective.

Water Saving Technology in California's Ag Land: A Coverage Comparison

Which Mobile Provider Has Better Coverage to Power Water Saving Technology in California's Ag Land?

In a July 5 article on water technology in the Los Angeles Times, addressed some of the challenges Silicon Valley is having in developing water saving technology for farms. Part of it is the difficulty in demonstrating value to farmers, part of it is a culture divide. "The next generation of precision agriculture — a world of wireless sensors, cloud-based data crunching, aerial imaging and app-based decision-making — may germinate in Silicon Valley, but it will have to take root here [California's Central Valley], where growers soak up about 80% of the water diverted for human use."

A more immediate problem might be mobile coverage. We did a recent analysis with the help of Cal State University Chico's Geographic Information Center of mobile coverage in California's agricultural areas. As we've published before in this blog, we relied on the mean minus 1 standard deviation values of test locations at 1,990 points across the state from the Fall 2014 field test. We then filtered for agricultural land using Classes 81 and 82 in the National Land Cover Database. Surprisingly, Verizon appears to have better coverage than AT&T in California's agricultural lands. One major caveat is that our field tests are performed only on roads, so it will take additional testing on agricultural lands, out in the fields, to get a more accurate picture of mobile coverage there.

Verizon Wireless Served Map (mean minus 1 standard deviation)

Legend:
Green: Greater than or equal to combined 6 megabits per second down and 1.5 up
Yellow: Below 6 megabits per second down and/or 1.5 megabits per second up
Red: Below 768 kilobits per second down and/or 200 kilobits per second up

 

AT&T Mobility Served Map (mean minus 1 standard deviation)

Fi Comes to CalSPEED

We were happy to see the first Fi tests run on CalSPEED last week. Fi is one of  Google's latest projects, and it combines Sprint, T-Mobile, and WiFi networks to provide seamless service. Click here for coverage of the April 2015 announcement by Google.

Here's a summary of last week's tests:

Backhaul Sleuthing in Rural California

As part of our investigation into why rural mobile performance is so much worse than urban, we had our testers run trace route tests to see where each provider sent traffic on its backhaul network. We just received the data output, and it's over 36 megabytes in comma delimited format. More than 16,000 tests, some with more than 20 "hops." It will take some time to decipher everything and draw conclusions, but one thing seems apparent immediately: AT&T has fewer hops than the other providers to get to the same destination. This may be one factor in AT&T having higher overall VoIP capability than the other providers (fewer hops, lower latency). Here is a table of the average number of hops to the West server for each provider:

Alpha testing of CalSPEED for iPhone begins now

We have just begun alpha testing the iPhone version of CalSPEED. The screen shot above, taken from an iPhone, shows that there's still work to do. For example, users will not need to enter a location ID (as our field testers have to do when we do mobile testing). The second image below is a draft version of the user interface that is being developed.

Spring 2015 Field Testing Complete!!

Our intrepid team of mobile field testers from Cal State Chico and Cal State Northridge have finished another round of mobile field testing, covering around 1,990 locations across the state. This is the first time we tested Android tablets in the field, and it significantly reduced testing time not having to swap out data cards on a netbook. Some of that time reduction, though, was taken up by a new traceroute test that the testers ran so we can begin to get a better look at the backhaul networks.

Results are still coming in and will be cleaned, compiled and analyzed over the coming months.

Thank you to everyone who helped make this a success!

- photo courtesy of Steven Crews

Broadband Testing At Mojave Wind Farm

This Vine reminds me of how California is leading the country in adopting renewable energy. California's Renewables Portfolio Standard (RPS) began in 2002 and is one of the most ambitious renewable energy standards in the country. The RPS program requires investor-owned utilities (IOUs), electric service providers, and community choice aggregators to increase procurement from eligible renewable energy resources to 33% of total procurement by 2020. Read more about RPS here.

Actual RPS Procurement Percentages in 2013

PG&E- 23.8 %

SCE – 21.6%

SDG&E - 23.6%

What Good Is A Coverage Map?

Have you ever gone to a mobile provider's web site or their store to see their coverage map? You'll notice the maps never show speeds. There's a reason: the maps are generally based on signal levels, not speed. Instead, providers say things like "4G LTE coverage," which means nothing to the average user. If you don't have time to read this all the way through, skip to #3 below.

As part of our state-wide mobile testing, we have started to peel back the layer of coverage and ask, "What does coverage really look like when we use actual speed?"

Here is a quick comparison of a single provider's coverage, 1) as advertised, 2) as measured, in terms of average speed, and 3) as adjusted by subtracting one standard deviation from the average (mean) speed.

1. Advertised Speed.
Here is one provider's maximum advertised downstream speed for California. The bright green indicates a downstream speed range of 10-25 megabits per second. The lighter green is 6-10 megabits per second, and the yellow represents a range from 768 kilobits per second to 1.5 megabits per second.

2. Measured Speed.
Here is the same provider's downstream speed based on test results at 1,990 locations across the state. Results from each location were spatially interpolated using a method called "kriging" to create a continuous surface. In this case, the bright green areas are significantly smaller, and more colors have appeared, representing slower speeds: 1.5-3 megabits per second (orange), and 3-6 megabits per second (brown).

3. Mean Minus One Standard Deviation.

Here, we adjusted the measured results at each of the 1,990 locations by subtracting the mean value by the standard deviation for the location where the measurement was performed. We do this, because we have observed that the higher the speed, the larger the standard deviation. In some cases, the standard deviation can be as high as 30% of the mean. This adjusted coverage map may likely be a more realistic picture of your mobile provider's coverage.

4. Higher Speed = Higher Variation
How large is the average standard deviation? Below is a chart showing how large the standard deviation can be when averaged across all measurements for urban, rural, and tribal. Graphs are for both upstream and downstream.

Upstream

Downstream

Reality Check: Sprint #1 Network in Two Years?

Sprint CEO Marcelo Claure predicted Sprint will have the top network in the next 18 to 24 months. Claure said that when he started, Sprint's network was "drop-dead last." We decided to see where Sprint has come from and how far it has to go to beat AT&T and Verizon in network performance. These data compare results from tests performed twice a year for smartphones and netbook data cards. The first three rounds of testing were done for 1,200 locations, the second three rounds expanded to 1,990 locations.

Device: Samsung Galaxy 4 (LTE)

1. Overall TCP Failure Rate

Looking at how often a TCP connection fails (TCP is the measurement used most often when comparing speed), data from tests of all four providers showed a slight increase in TCP failure, with T-Mobile and Sprint on the higher end.

2. Locations Where TCP Met "Served" Threshold

For purposes of comparison, we looked at the percentage of tested locations where average TCP downstream met or exceeded 6 megabits per second and average upstream met or exceeded 1.5 megabits per second. This threshold is what we use at the CPUC for determining "served" status. Note that these averages are not adjusted using standard deviation. "Average" in this case averages smartphone & netbook as well as west and east coast servers.

3. Over The Top VoIP Readiness

Using a Mean Opinion Score formula we reported on earlier in this blog, we calculated the number of measurements (1 measurement = 1 location/device/provider combination) where the calculated Mean Opinion Score met or exceeded 4.0.

Round 7 Testing 70% Complete

New tester record: 30 sites in a day

Here is where Tester #2 finished his day yesterday with 30 test locations under his belt:

Here are the smartphone results from that location:

• AT&T Mobility: 2.297 Mb/s up, 16.592 Mb/s down
• Sprint: No service
• T-Mobile: 4.368 Mb/s up, 4.354 Mb/s down
• Verizon Wireless: 1.544 Mb/s up, 7.596 Mb/s down

Five Dollar A Gallon Gas? No Other Choice...

Sand Storm in Death Valley

How To Estimate VoIP Quality? Mean Opinion Score (MOS)

Some of you have asked us how we calculate a mean opinion score to estimate voice quality for over-the-top voice over internet protocol (VoIP) service. Well, here is the formula we use.

MOS

=IF(AND(ISNUMBER(R-value)),IF(AND(R-value>0,R-value<101),(1+0.035*R-value+0.000007*R-value*(R-value-60)*(100-R-value)),0),0)

R-value

=IF(Effective latency<160,(93.2-(Effective latency/40)-2.5*Avg. UDP packet loss),(93.2-(Effective latency-120)/10-2.5*Avg. UDP packet loss))

Effective latency

=Avg. latency + Avg. jitter*2+10

Easter Island Visitation?

Mysterious discoveries on the road....

Mobile Testing now in the Second Half

One this is for sure: switching from netbooks with data cards over to tablets has decreased testing time. As of yesterday, our team had completed over half of the test locations. Keep up the good work!

Thanksgiving in May

Spring 2015 Testing More Than One Fifth Complete