New to CalSPEED? Frequently Answered Questions

     Those of you new to CalSPEED often ask how it compares to other speed testing apps. Here are answers to some of the more common questions.     

      

      How Did CalSPEED Begin? CalSPEED was originally funded by a State Broadband Initiative grant from the National Telecommunications and Information Administration. The testing program began in spring 2012 and has now completed 10 rounds. The test program collects not only speed data, but also speed variation, latency, jitter, and packet loss. With these data, we are able to estimate performance for “over-the-top” streaming voice and video service. With ten rounds of semi-annual mobile testing completed, the CPUC has one of the largest public data sets of mobile broadband performance.

      Where Are Tests Performed? The CPUC tests the same 1,990 locations twice a year. The breakdown is 37% urban locations, 56% rural locations, and 7% tribal locations, which were randomly generated. The field test relies on two devices from four major providers (AT&T, Sprint, T-Mobile, and Verizon). 80 TCP tests are performed for each provider, on each device, at each of the 1,990 locations.

      Is 1,990 Locations Enough? Using advanced geo-statistical methods, we are able to  interpolate service characteristics likely to be experienced by a user located anywhere in the state. The CPUC designed the mix of test locations to cover not only urban places where people live and work, but also rural locations where people may be passing through, such as rural highways and state and national parks. All tests are performed along roads navigable by automobile.

      Why Not Test In Every Census Block? Performing field tests in all of California’s 710,145 census blocks would be prohibitively expensive, impractical and unnecessary. For this reason, neither the CPUC nor mobile providers like Verizon, AT&T, T-Mobile, and Sprint perform tests in every census block, but instead use statistical techniques to approximate service characteristics in between tested locations.

     

 Why Two Servers? Most testing applications use only one, generally nearby, server.  This method understates latency and overstates throughput as compared to using multiple, geographically-diverse servers.Testing to a nearby server results in speeds likely to be experienced for applications such as streaming movies, where content is often cached locally due to its popularity.  However, much of the content broadband users access is not cached locally, so CalSPEED tests two two servers -- one in Arlington, Virginia, the other in San Jose, California -- to understand the role of back haul networks in each provider’s delivery of mobile broadband. While using more than two test servers in disparate locations across the globe would be desirable, using both east coast and west coast servers yields more representative results that testing to only one server.

Is This Better Than SpeedTest.net or the FCC's Speed Test? As shown in a study published by Novarum in 2014 comparing Ookla, FCC, and CalSPEED testing applications, results for Ookla and FCC tests tend to be higher because both intentionally select test servers for lowest latency, which tend to be geographically closer. Moreover, Ookla’s test further biases results by discarding the bottom half of upstream results and bottom third of downstream results. By consistently testing to the same two servers, one on each coast of the continent, CalSPEED provides a reliable backhaul performance metric for each of the four mobile providers. Since we began testing in 2012, we have seen the performance (latency) difference between east and west servers decrease.
 
How Else Does CalSPEED Differ From Other Speed Tests? Most speed test applications rely on crowd sourcing. Crowd sourcing has an inherent selection bias of only collecting data from where it is chosen to be used. Where data is collected, it is biased towards who collected it, why, when and where. In contrast, the CalSPEED methodology has testers return to the same location every time, and the geographic distribution of test locations provides a more complete picture of mobile broadband across the state.

How Many TCP Threads Does CalSPEED Use?  Multi-threading means opening more than one connection to the host and combining them in order to boost overall throughput and is used by many speed test applications. When the CPUC designed CalSPEED, we examined the effect of using multiple threads (“flows”) and concluded there was no material difference in mobile throughput between four threads versus eight threads or sixteen threads. The current test design has 4 threads, each divided into ten 1-second tests for upstream to the west server, then again to the east server. The same is true for downstream. This is then repeated a second time, totaling eighty 1-second tests. Most applications only use one thread. 

Why Do Speed Testing? Carriers Already Have Coverage Maps. Most provider maps show a single coverage color and say things like "4G/LTE Coverage." Through CalSPEED, the CPUC has been able to discern more subtle speed and coverage differences by region. Some providers advertise speeds, but we have observed that those speeds are not ubiquitous, that is, they are not available everywhere providers claim to offer service. Speeds vary widely depending on if you are in an urban, rural, or tribal location. For this reason, we create a heat map of speeds based on actual field test data, and the heat map shows how speeds vary across the state. 

Why Not Use Average Speed, Like Mean or Median? The CPUC has demonstrated[1] through years of methodical field testing that mean and median speeds, by themselves, are unreliable indicators of what consumers can expect to experience reliably at a location. CalSPEED takes observed variability into account to determine speeds that consumers can consistently expect to receive.  As mean throughput increases, so does the amount of variability around the mean.

---

[1] See Section 2.4 Intra-Session Variation in “CalSPEED: California Mobile Broadband - An Assessment - Fall 2014,” by Novarum.

Complete! Fall 2016 Mobile Field Testing

Thank you to the field test drivers from Cal State University, Chico and Cal State University Northridge for finishing this 10th round of mobile field testing. Thanks also to CSU Chico's Geographical Information Center and CSU Monterey Bay's School of Computing and Design for their continued support for this innovative project.

As always, we will be posting summary data on the CPUC's website once it becomes available.

Hello Wireless Loops. Goodbye Fiber? (Part 2)

Last week, I posted three scenarios examining AT&T's mobile LTE coverage of high cost-eligible homes under the FCC's Connect America Fund Phase II program. That analysis assumed AT&T would be using their existing mobile network and licensed LTE spectrum...

Turns out, that was a rosy assumption.

Since last Friday, I received more information from Steve Blum about AT&T's plans to deploy wireless local loops (thanks, Steve!). Rather than revisit the issues, I want to make some corrections and comments to my previous post.

1. AT&T plans to deploy a new technology for wireless local loops whose "success in the marketplace is thus unproven." (paragraph 51 of their SEC 425 filing). For more, see #1 below.
2. Rather than use licensed LTE spectrum, which includes the incredibly valuable 700 MHz band, it appears AT&T will be using a higher frequency (2,300 MHz, or 2.3 GHz), which requires line of sight between cell tower and customer location. For more, see #2 below.
3. Other risks include spectrum constraints on the number of subscribers who can use the service simultaneously during peak usage, and this "reduce[s] the prospects for a successful rollout by AT&T of fixed WLL as a standalone product." For more, see #3 below.

1. Did AT&T Say They'll Need More Cell Sites? 

Group President and Chief Strategy Officer John Stankey's testimony, which is part of AT&T's Form 425 filing for the DIRECTV acquisition, states: "A fixed [wireless local loop] service requires substantial upfront investments. AT&T must install additional antennas and other equipment at each cell site in areas it seeks to serve." (paragraph 50). 

There's no mention of needing more cell sites. 

Modifying existing cell sites is one thing. Acquiring, permitting, and building new sites is completely different. As one El Dorado County supervisor says, new sites are very controversial because of their "aesthetic downside." Aside from having to find property and possibly develop it, there's the sticky problem of getting community buy-in for more cell towers. 

If AT&T plans to use a higher frequency like 2,300 MHz for wireless, they will need to add cell sites. That takes time. Finding additional sites, much less building new ones, will have a significant impact on AT&T's ability to deploy wireless local loop services in a timely manner. It also puts more pressure on local governments to expedite site approvals. 

How many new sites are needed? To get an idea, here is AT&T's presentation to the El Dorado County Board of Supervisors. AT&T estimates they'll need to modify 4 cell towers and add 34 new locations -- either collocated on existing buildings or newly constructed cell sites. 38 locations seems insufficient to serve all of El Dorado County’s high cost-eligible CAF II households, so maybe this is the first phase in AT&T’s five-year rollout. 

2. Frequency Matters 

There are solid engineering arguments why 700 MHz is more valuable than 2,300 MHz for delivering wireless broadband to homes. I’ll address only the line of sight argument. The larger-wavelength 700 MHz frequency penetrates buildings and foliage better than the shorter-wavelength 2,300 MHz frequency. For 2,300 MHz service to reliably deliver broadband, there needs to be line of sight, meaning you can physically see, without obstruction, between cell tower antenna and customer antenna. Without line of sight, service will be less reliable than with 700 MHz. Many parts of rural California are mountainous and have trees, which makes line of sight broadband difficult and expensive to deploy.

Home in El Dorado County with Trees Obstructing Line of Sight

  

Diagram showing line of sight wireless broadband. No obstructions allowed.

Diagram showing non-line of sight wireless broadband, e.g. 700 MHz

 

Credit: L-Com Connectivity, www.L-com.com 

3. Is Wireless Local Loop Service Viable Without Video? 

AT&T’s Form 425 testimony mentions the importance to their revenue stream of bundling broadband (through WLL, U-Verse, or their fiber product, GigaPower) with video (through DIRECTV), voice, and even mobile. As John Stankey stated, capacity constraints reduce the prospects of WLL as a viable standalone service. Put another way, WLL might be financially viable only if a subscriber opts to subscribe to video and/or mobile as well.

The prospect of adding new satellite and cable television subscriptions is dubious in today’s marketplace. One study claims that more than 40% of satellite and cable subscribers are planning on cutting back or dropping their pay TV service. Broadband data from the CPUC's 2013 annual report on the Digital Infrastructure and Video Competition Act indicates that California's state video franchise holders now provide more broadband service than video service.

That doesn’t bode well for AT&T’s DIRECTV subscribers, some of whom may be switching to over-the-top services like Netflix or Amazon Prime and require only a reliable internet connection to watch shows. To underscore that point, AT&T's Form 425 filing for the DIRECTV acquisition includes an economic assessment, which states "higher Internet access speeds will erode the traditional advantage that DIRECTV's satellite network has given it" over video on demand providers like Netflix. (page 54)

More on AT&T’s proposed WLL broadband service (from the El Dorado County video):

• Broadband speeds must meet or exceed 10 megabits per second down and 1 up
• Latency should not exceed 100 milliseconds
• Initial minimum usage allowance of 150 gigabytes per month
• Monthly rates must be comparable to fixed wireline urban rates for the same level of service in urban areas or at or below the FCC Annual National Rate Benchmark (approx. $72/month)
• AT&T must meet specified deployment milestone dates by state
• 40% built by year-end 2017
• 60% by year-end 2018
• 80% by year-end 2019
• 100% by year-end 2020-21

Week 4 Testing: 82% Complete

Our testers are nearly finished with the 10th round of statewide field testing with 82% complete as of last Friday. Here's a map of the completed locations in red, and those still needing to be tested in blue: