Connect America Fund Layer Now Available on CA Interactive BB Map

As a follow up to my November 1 post on AT&T's plans to deploy wireless local loops as part of their obligations under Connect America Fund Phase II (see "Hello Wireless Loops. Goodbye Fiber? Part 2"), the Connect America Fund Phase II eligible areas are now loaded onto the California Interactive Broadband Map. As shown in the legend to the left, four carriers' territories are shown in light blue (AT&T), dark blue (Consolidated), gray (Frontier), and red (Verizon, now also Frontier). To view the areas, go to the map's menu on the right and expand the FCC Data menu. Check the box next to "Connect America Fund Phase II Locations."

Connect America Fund Phase II Eligible Census Blocks by Carrier

CalSPEED iPhone Ver 1.1.1 Now Available on iTunes

 We've made some improvements to CalSPEED with the iPhone version 1.1.1. Changes include:

• Clearer indoor/outdoor slider
• WiFi reminder screen if device is connected to WiFi
• California and Virginia server testing segments indicated at bottom 
• Streaming video quality indicator added to results summary
• Results history shows video streaming and Mean Opinion Score (MOS) estimate for over the top service

New to CalSPEED? Frequently Answered Questions

     Those of you new to CalSPEED often ask how it compares to other speed testing apps. Here are answers to some of the more common questions.     

      

      How Did CalSPEED Begin? CalSPEED was originally funded by a State Broadband Initiative grant from the National Telecommunications and Information Administration. The testing program began in spring 2012 and has now completed 10 rounds. The test program collects not only speed data, but also speed variation, latency, jitter, and packet loss. With these data, we are able to estimate performance for “over-the-top” streaming voice and video service. With ten rounds of semi-annual mobile testing completed, the CPUC has one of the largest public data sets of mobile broadband performance.

      Where Are Tests Performed? The CPUC tests the same 1,990 locations twice a year. The breakdown is 37% urban locations, 56% rural locations, and 7% tribal locations, which were randomly generated. The field test relies on two devices from four major providers (AT&T, Sprint, T-Mobile, and Verizon). 80 TCP tests are performed for each provider, on each device, at each of the 1,990 locations.

      Is 1,990 Locations Enough? Using advanced geo-statistical methods, we are able to  interpolate service characteristics likely to be experienced by a user located anywhere in the state. The CPUC designed the mix of test locations to cover not only urban places where people live and work, but also rural locations where people may be passing through, such as rural highways and state and national parks. All tests are performed along roads navigable by automobile.

      Why Not Test In Every Census Block? Performing field tests in all of California’s 710,145 census blocks would be prohibitively expensive, impractical and unnecessary. For this reason, neither the CPUC nor mobile providers like Verizon, AT&T, T-Mobile, and Sprint perform tests in every census block, but instead use statistical techniques to approximate service characteristics in between tested locations.

     

 Why Two Servers? Most testing applications use only one, generally nearby, server.  This method understates latency and overstates throughput as compared to using multiple, geographically-diverse servers.Testing to a nearby server results in speeds likely to be experienced for applications such as streaming movies, where content is often cached locally due to its popularity.  However, much of the content broadband users access is not cached locally, so CalSPEED tests two two servers -- one in Arlington, Virginia, the other in San Jose, California -- to understand the role of back haul networks in each provider’s delivery of mobile broadband. While using more than two test servers in disparate locations across the globe would be desirable, using both east coast and west coast servers yields more representative results that testing to only one server.

Is This Better Than SpeedTest.net or the FCC's Speed Test? As shown in a study published by Novarum in 2014 comparing Ookla, FCC, and CalSPEED testing applications, results for Ookla and FCC tests tend to be higher because both intentionally select test servers for lowest latency, which tend to be geographically closer. Moreover, Ookla’s test further biases results by discarding the bottom half of upstream results and bottom third of downstream results. By consistently testing to the same two servers, one on each coast of the continent, CalSPEED provides a reliable backhaul performance metric for each of the four mobile providers. Since we began testing in 2012, we have seen the performance (latency) difference between east and west servers decrease.
 
How Else Does CalSPEED Differ From Other Speed Tests? Most speed test applications rely on crowd sourcing. Crowd sourcing has an inherent selection bias of only collecting data from where it is chosen to be used. Where data is collected, it is biased towards who collected it, why, when and where. In contrast, the CalSPEED methodology has testers return to the same location every time, and the geographic distribution of test locations provides a more complete picture of mobile broadband across the state.

How Many TCP Threads Does CalSPEED Use?  Multi-threading means opening more than one connection to the host and combining them in order to boost overall throughput and is used by many speed test applications. When the CPUC designed CalSPEED, we examined the effect of using multiple threads (“flows”) and concluded there was no material difference in mobile throughput between four threads versus eight threads or sixteen threads. The current test design has 4 threads, each divided into ten 1-second tests for upstream to the west server, then again to the east server. The same is true for downstream. This is then repeated a second time, totaling eighty 1-second tests. Most applications only use one thread. 

Why Do Speed Testing? Carriers Already Have Coverage Maps. Most provider maps show a single coverage color and say things like "4G/LTE Coverage." Through CalSPEED, the CPUC has been able to discern more subtle speed and coverage differences by region. Some providers advertise speeds, but we have observed that those speeds are not ubiquitous, that is, they are not available everywhere providers claim to offer service. Speeds vary widely depending on if you are in an urban, rural, or tribal location. For this reason, we create a heat map of speeds based on actual field test data, and the heat map shows how speeds vary across the state. 

Why Not Use Average Speed, Like Mean or Median? The CPUC has demonstrated[1] through years of methodical field testing that mean and median speeds, by themselves, are unreliable indicators of what consumers can expect to experience reliably at a location. CalSPEED takes observed variability into account to determine speeds that consumers can consistently expect to receive.  As mean throughput increases, so does the amount of variability around the mean.

---

[1] See Section 2.4 Intra-Session Variation in “CalSPEED: California Mobile Broadband - An Assessment - Fall 2014,” by Novarum.

Complete! Fall 2016 Mobile Field Testing

Thank you to the field test drivers from Cal State University, Chico and Cal State University Northridge for finishing this 10th round of mobile field testing. Thanks also to CSU Chico's Geographical Information Center and CSU Monterey Bay's School of Computing and Design for their continued support for this innovative project.

As always, we will be posting summary data on the CPUC's website once it becomes available.

Hello Wireless Loops. Goodbye Fiber? (Part 2)

Last week, I posted three scenarios examining AT&T's mobile LTE coverage of high cost-eligible homes under the FCC's Connect America Fund Phase II program. That analysis assumed AT&T would be using their existing mobile network and licensed LTE spectrum...

Turns out, that was a rosy assumption.

Since last Friday, I received more information from Steve Blum about AT&T's plans to deploy wireless local loops (thanks, Steve!). Rather than revisit the issues, I want to make some corrections and comments to my previous post.

1. AT&T plans to deploy a new technology for wireless local loops whose "success in the marketplace is thus unproven." (paragraph 51 of their SEC 425 filing). For more, see #1 below.
2. Rather than use licensed LTE spectrum, which includes the incredibly valuable 700 MHz band, it appears AT&T will be using a higher frequency (2,300 MHz, or 2.3 GHz), which requires line of sight between cell tower and customer location. For more, see #2 below.
3. Other risks include spectrum constraints on the number of subscribers who can use the service simultaneously during peak usage, and this "reduce[s] the prospects for a successful rollout by AT&T of fixed WLL as a standalone product." For more, see #3 below.

1. Did AT&T Say They'll Need More Cell Sites? 

Group President and Chief Strategy Officer John Stankey's testimony, which is part of AT&T's Form 425 filing for the DIRECTV acquisition, states: "A fixed [wireless local loop] service requires substantial upfront investments. AT&T must install additional antennas and other equipment at each cell site in areas it seeks to serve." (paragraph 50). 

There's no mention of needing more cell sites. 

Modifying existing cell sites is one thing. Acquiring, permitting, and building new sites is completely different. As one El Dorado County supervisor says, new sites are very controversial because of their "aesthetic downside." Aside from having to find property and possibly develop it, there's the sticky problem of getting community buy-in for more cell towers. 

If AT&T plans to use a higher frequency like 2,300 MHz for wireless, they will need to add cell sites. That takes time. Finding additional sites, much less building new ones, will have a significant impact on AT&T's ability to deploy wireless local loop services in a timely manner. It also puts more pressure on local governments to expedite site approvals. 

How many new sites are needed? To get an idea, here is AT&T's presentation to the El Dorado County Board of Supervisors. AT&T estimates they'll need to modify 4 cell towers and add 34 new locations -- either collocated on existing buildings or newly constructed cell sites. 38 locations seems insufficient to serve all of El Dorado County’s high cost-eligible CAF II households, so maybe this is the first phase in AT&T’s five-year rollout. 

2. Frequency Matters 

There are solid engineering arguments why 700 MHz is more valuable than 2,300 MHz for delivering wireless broadband to homes. I’ll address only the line of sight argument. The larger-wavelength 700 MHz frequency penetrates buildings and foliage better than the shorter-wavelength 2,300 MHz frequency. For 2,300 MHz service to reliably deliver broadband, there needs to be line of sight, meaning you can physically see, without obstruction, between cell tower antenna and customer antenna. Without line of sight, service will be less reliable than with 700 MHz. Many parts of rural California are mountainous and have trees, which makes line of sight broadband difficult and expensive to deploy.

Home in El Dorado County with Trees Obstructing Line of Sight

  

Diagram showing line of sight wireless broadband. No obstructions allowed.

Diagram showing non-line of sight wireless broadband, e.g. 700 MHz

 

Credit: L-Com Connectivity, www.L-com.com 

3. Is Wireless Local Loop Service Viable Without Video? 

AT&T’s Form 425 testimony mentions the importance to their revenue stream of bundling broadband (through WLL, U-Verse, or their fiber product, GigaPower) with video (through DIRECTV), voice, and even mobile. As John Stankey stated, capacity constraints reduce the prospects of WLL as a viable standalone service. Put another way, WLL might be financially viable only if a subscriber opts to subscribe to video and/or mobile as well.

The prospect of adding new satellite and cable television subscriptions is dubious in today’s marketplace. One study claims that more than 40% of satellite and cable subscribers are planning on cutting back or dropping their pay TV service. Broadband data from the CPUC's 2013 annual report on the Digital Infrastructure and Video Competition Act indicates that California's state video franchise holders now provide more broadband service than video service.

That doesn’t bode well for AT&T’s DIRECTV subscribers, some of whom may be switching to over-the-top services like Netflix or Amazon Prime and require only a reliable internet connection to watch shows. To underscore that point, AT&T's Form 425 filing for the DIRECTV acquisition includes an economic assessment, which states "higher Internet access speeds will erode the traditional advantage that DIRECTV's satellite network has given it" over video on demand providers like Netflix. (page 54)

More on AT&T’s proposed WLL broadband service (from the El Dorado County video):

• Broadband speeds must meet or exceed 10 megabits per second down and 1 up
• Latency should not exceed 100 milliseconds
• Initial minimum usage allowance of 150 gigabytes per month
• Monthly rates must be comparable to fixed wireline urban rates for the same level of service in urban areas or at or below the FCC Annual National Rate Benchmark (approx. $72/month)
• AT&T must meet specified deployment milestone dates by state
• 40% built by year-end 2017
• 60% by year-end 2018
• 80% by year-end 2019
• 100% by year-end 2020-21

Week 4 Testing: 82% Complete

Our testers are nearly finished with the 10th round of statewide field testing with 82% complete as of last Friday. Here's a map of the completed locations in red, and those still needing to be tested in blue:

CalSPEED Testing @ Head of the Charles Regatta

Last weekend, River City Rowing Club teammates and I competed in the Senior Master Four race at the annual Head of the Charles Regatta in Boston. While there, I ran several speed tests at the AirBnB where we stayed -- both on the WiFi provided and on AT&T's LTE network. Here are the results:

Head of the Charles racecourse. Test location shown in red point below.

River City Rowing Club Sr. Master Four: Michael Paiva, Dan Tharp, Paul Crawford, Whitney Powell, me

Week 3 Mobile Testing Progress - 75% complete as of Oct. 26

 

Hello Wireless Loops. Goodbye Fiber? (Part 1)

Wireless Loops As Copper Substitute
There have been reports that California's largest local exchange carrier and Connect America Fund Phase II recipient, AT&T California (maps of California's CAF II areas), plans to use wireless loops to deliver broadband to high cost-eligible households -- in other words, use mobile (i.e. "wireless loops") as a substitute for copper telephone lines to offer speeds of 10 megabits per second down and 1 megabit per second up or faster (10/1). How the FCC measures AT&T's progress in deploying wireless 10/1 could greatly influence the level of investment AT&T needs to make. This post analyzes several scenarios where AT&T uses mobile LTE spectrum to deliver wireless loops. Read Part 2 for corrections and comments to this analysis.
(photo credit: Owen Rochte)

Background
Last year, AT&T, Verizon, Frontier, and Consolidated agreed to receive from the FCC, collectively, $105 million annually for five years to upgrade California homes so they can get at least a 10/1 connection to the internet. The federal program that pays for this is called Connect America Fund Phase II (CAF II), and it is funded by surcharges on our telephone bills.

Fiber Optic Reality Check
As we enter the last quarter of 2016, it's time for a reality check. For those of you living in CAF II high cost areas (primarily rural California) who've been holding out hope for fiber optic broadband access to your home, you may be waiting in vain. That's because CAF II doesn't pay enough for fiber to the home, and carriers are not obligated to build fiber to the home. Furthermore, once the CPUC's California Advanced Services Fund runs out of money, there's arguably no funding mechanism, either federal or state, to bring fiber to rural California homes. 

Can Mobile Deliver?
If engineered as a wireline substitute, mobile can deliver 10 megabits per second downstream and 1 megabit per second upstream (10/1) to the home. The question is what will be used as proof that AT&T's 141,000 CAF II-eligible homes have been upgraded to receive that level of service? We know from mobile field tests that mobile speeds are highly variable from one moment to the next, so in order to deliver a reliable 10/1 service, AT&T may need to augment their network to deliver considerably faster speeds on average. Refer to the CPUC's comments to the FCC on GN Docket No. 15-191 regarding the highly variable nature of mobile broadband and the importance of accounting for reliability when measuring broadband speeds.

The "LTE" Indicator On Your Phone Doesn't Mean Fast Speeds
The FCC has made available to the public some of AT&T's mobile broadband deployment data. Unfortunately, speed data are missing. What remains is a file showing areas with access to LTE (4G "Long Term Evolution") service, as well as other, lower levels of service. We know from experience that mobile coverage shrinks as speeds increase, so it's impossible to tell how large a 10/1 service area extends across California.

Three CAF II Wireless Loop Scenarios 
If AT&T is going to use LTE wireless loops (i.e. home broadband access over the mobile network), here are three initial takes on how many CAF II-eligible households are theoretically covered today using various data sets. Based on these scenarios, AT&T has to make either very little investment to offer 10/1 service to their CAF II high cost households, or they have to make a significant investment. The answer hinges on what validation method the FCC uses to ensure AT&T has met its CAF II obligations to offer 10/1 or faster.

• Scenario 1: FCC 477 Deployment Data - Using the December 2015 deployment data AT&T filed with the FCC, 85% of AT&T's CAF II high cost households have LTE coverage. Note that "LTE" is not the same as 10/1 or faster. However, if the FCC were to use 477 data as proof of meeting CAF II obligations, AT&T could conceivably claim subsidies for most of the 141,000 high cost households without doing any network upgrades.
• Scenario 2: Interpolated Average Speed Data - Based on interpolated average speeds from Spring 2016 mobile field testing, over half of AT&T's CAF II high cost households appear to have access to speeds of 10/1 or faster.
• Scenario 3: Interpolated Mean Minus 2 Standard Deviations - Using this stricter standard, where we reduce the average speed at each of the 1,990 field test locations by two standard deviations and create an interpolated surface using those adjusted speeds, nearly all of the 141,000 households appear to remain CAF II eligible. This suggests that AT&T would need to make a significant investment in their mobile network to deliver a reliable 10/1 or faster service to their high cost households.

Conclusion
Of these three scenarios, the answer to how much additional coverage is necessary is likely to be somewhere between scenarios 2 and 3, because scenario 1 is does not rely on a speed threshold, but rather simply the presence of LTE service. "LTE" offers a wide range of speeds, including zero kilobits per second. For the sake of reliability, the FCC should implement a robust mobile measurement program, such as CalSPEED, to verify AT&T has met its CAF II obligations using wireless loops.

Scenario 1: FCC 477 Deployment Data: 85% LTE Coverage

  
Scenario 2: Interpolated Average Speed Data: 57% High Cost Households with 10/1 or Greater

Scenario 3: Interpolated Mean Minus 2 Standard Deviations: No Households with 10/1
 

Here is what AT&T's LTE coverage looks like based on Scenario 1 (no speeds indicated):

Here is what AT&T's LTE coverage looks like based on Scenario 3 (speeds indicated in legend):

Roadrunner & Wile E. Coyote

This week, we were treated to an amazing photo by tester Steve Crews of a roadrunner with mountains in the background. The team finished Week 2 having completed 46% of the 1,990 locations (see map image below, completed location shown in red).

Average speeds shown in table below are in kilobits per second (Kbps), so divide by 1,000 to get megabits per second (Mbps).

 

Two CalSPEED Locations in Destroyed City of Homs, Syria

As I've said in earlier posts, back when we launched CalSPEED in 2013, we didn't restrict distribution to only the United States. We added the restriction because we didn't want people running tests all over the world. CalSPEED is designed to be used by California residents so they can understand how their mobile provider is performing. Nevertheless, we occasionally still see tests appearing in unexpected parts of the world.

At the end of 2014, there were three tests performed at geographic coordinates that appear to be in the City of Homs, Syria. A quick look at those locations in Google Earth reveals the destruction of that city from years of war. See before-and-after photos in The Guardian here.

A New York Times' article on the destruction of another Syrian city, Aleppo, estimates the number of displaced people around the world is equivalent in number to the entire population of the United Kingdom or France. It would seem that number includes all displaced people, not just Syrians, but these images of Syria, even from Google Earth, are devastating.

Here's what part of Homs looks like from Google Earth:

Believe Me, It's Radar Enforced!

My favorite is the "Radar Enforced" road sign collection. Product of a frustrated highway patrol officer? Below the nine photo collage is a cryptic fortune cookie message.

Week 1 Progress Map

The red dots indicate which locations our testers have completed as of the end of last week. The blue dots are locations that will be tested this week and in coming weeks.

Bravo! Photos from Week 1

I am constantly amazed at the geographic diversity of California, and the photos coming in from our field testers are proof. Their photos are wonderful. I'm adding a few of them from Week 1, but feel free to follow their progress on Instagram at cpuc_broadband_testing.