New to CalSPEED? Frequently Answered Questions

     Those of you new to CalSPEED often ask how it compares to other speed testing apps. Here are answers to some of the more common questions.     

      

      How Did CalSPEED Begin? CalSPEED was originally funded by a State Broadband Initiative grant from the National Telecommunications and Information Administration. The testing program began in spring 2012 and has now completed 10 rounds. The test program collects not only speed data, but also speed variation, latency, jitter, and packet loss. With these data, we are able to estimate performance for “over-the-top” streaming voice and video service. With ten rounds of semi-annual mobile testing completed, the CPUC has one of the largest public data sets of mobile broadband performance.

      Where Are Tests Performed? The CPUC tests the same 1,990 locations twice a year. The breakdown is 37% urban locations, 56% rural locations, and 7% tribal locations, which were randomly generated. The field test relies on two devices from four major providers (AT&T, Sprint, T-Mobile, and Verizon). 80 TCP tests are performed for each provider, on each device, at each of the 1,990 locations.

      Is 1,990 Locations Enough? Using advanced geo-statistical methods, we are able to  interpolate service characteristics likely to be experienced by a user located anywhere in the state. The CPUC designed the mix of test locations to cover not only urban places where people live and work, but also rural locations where people may be passing through, such as rural highways and state and national parks. All tests are performed along roads navigable by automobile.

      Why Not Test In Every Census Block? Performing field tests in all of California’s 710,145 census blocks would be prohibitively expensive, impractical and unnecessary. For this reason, neither the CPUC nor mobile providers like Verizon, AT&T, T-Mobile, and Sprint perform tests in every census block, but instead use statistical techniques to approximate service characteristics in between tested locations.

     

 Why Two Servers? Most testing applications use only one, generally nearby, server.  This method understates latency and overstates throughput as compared to using multiple, geographically-diverse servers.Testing to a nearby server results in speeds likely to be experienced for applications such as streaming movies, where content is often cached locally due to its popularity.  However, much of the content broadband users access is not cached locally, so CalSPEED tests two two servers -- one in Arlington, Virginia, the other in San Jose, California -- to understand the role of back haul networks in each provider’s delivery of mobile broadband. While using more than two test servers in disparate locations across the globe would be desirable, using both east coast and west coast servers yields more representative results that testing to only one server.

Is This Better Than SpeedTest.net or the FCC's Speed Test? As shown in a study published by Novarum in 2014 comparing Ookla, FCC, and CalSPEED testing applications, results for Ookla and FCC tests tend to be higher because both intentionally select test servers for lowest latency, which tend to be geographically closer. Moreover, Ookla’s test further biases results by discarding the bottom half of upstream results and bottom third of downstream results. By consistently testing to the same two servers, one on each coast of the continent, CalSPEED provides a reliable backhaul performance metric for each of the four mobile providers. Since we began testing in 2012, we have seen the performance (latency) difference between east and west servers decrease.
 
How Else Does CalSPEED Differ From Other Speed Tests? Most speed test applications rely on crowd sourcing. Crowd sourcing has an inherent selection bias of only collecting data from where it is chosen to be used. Where data is collected, it is biased towards who collected it, why, when and where. In contrast, the CalSPEED methodology has testers return to the same location every time, and the geographic distribution of test locations provides a more complete picture of mobile broadband across the state.

How Many TCP Threads Does CalSPEED Use?  Multi-threading means opening more than one connection to the host and combining them in order to boost overall throughput and is used by many speed test applications. When the CPUC designed CalSPEED, we examined the effect of using multiple threads (“flows”) and concluded there was no material difference in mobile throughput between four threads versus eight threads or sixteen threads. The current test design has 4 threads, each divided into ten 1-second tests for upstream to the west server, then again to the east server. The same is true for downstream. This is then repeated a second time, totaling eighty 1-second tests. Most applications only use one thread. 

Why Do Speed Testing? Carriers Already Have Coverage Maps. Most provider maps show a single coverage color and say things like "4G/LTE Coverage." Through CalSPEED, the CPUC has been able to discern more subtle speed and coverage differences by region. Some providers advertise speeds, but we have observed that those speeds are not ubiquitous, that is, they are not available everywhere providers claim to offer service. Speeds vary widely depending on if you are in an urban, rural, or tribal location. For this reason, we create a heat map of speeds based on actual field test data, and the heat map shows how speeds vary across the state. 

Why Not Use Average Speed, Like Mean or Median? The CPUC has demonstrated[1] through years of methodical field testing that mean and median speeds, by themselves, are unreliable indicators of what consumers can expect to experience reliably at a location. CalSPEED takes observed variability into account to determine speeds that consumers can consistently expect to receive.  As mean throughput increases, so does the amount of variability around the mean.

---

[1] See Section 2.4 Intra-Session Variation in “CalSPEED: California Mobile Broadband - An Assessment - Fall 2014,” by Novarum.