Big Telcos and Broadband

A recent article in Telecompetitor reports that analysts at Moffett Nathanson expect the big telcos to start making inroads into the near-monopoly for broadband currently enjoyed by the cable companies. The article focused specifically on AT&T, but some other big telcos like CenturyLink are also aggressively expanding fiber networks.

I would have to assume that the analysts got the following goals directly from AT&T because I can’t find any other references to these specific goals. But each of these is in line with statements made by AT&T executives over the last year. According to the article, AT&T broadband goals over the next few years are as follows:

  • Offer broadband speeds below 50 Mbps to 30 million passings using DSL;
  • Offer broadband speeds between 50 – 100 Mbps to 20 million passings using paired copper VDSL;
  • Offer ‘near gigabit’ speeds to 10 million passings using via 5G wireless;
  • Offer gigabit speeds using FTTH technology to 14 million residential passings and 8 million business passings.

The real news here is in the last two bullet points. AT&T accepted the goal from the FCC for passing 12.5 million customers with FTTH from the merger with DirecTV. It’s big news if they intend to extend that to 22 million passings. And the goal of using millimeter wave radios to reach another 10 million potential customers is something new.

If AT&T meets these goals they will be bringing serious competition to the cable companies. AT&T and the other telcos have been bleeding DSL customers for over a decade and handed the cable companies a near-monopoly on fast broadband in most urban and suburban markets. According to Moffett Nathanson the telco expansion will bring near-gigabit speeds on telco networks to 32% of the country.

It’s important to understand where the new AT&T broadband is being built. The majority of the new coverage is in three market niches – apartment buildings, new greenfield housing developments and business districts. AT&T’s expansion has largely focused on these specific market niches and is likely to continue to do so. AT&T is not proposing to duplicate what Verizon did with its FiOS network and bring broadband to older single family home neighborhoods. They are instead focusing on buildouts where the the cost of construction per customer is the lowest – the ultimate cherry-picking network.

This means that the AT&T coverage will bring the opportunity for gigabit broadband to a much larger footprint, but that’s not always going to bring customer choice. In the MDU market many landlords are still allowing only one ISP into their apartment complexes. As telcos like AT&T compete with the cable companies for this market the broadband speeds in apartments and condos will get much faster, but many customers will still only have the option to buy from whatever ISP that landlord has allowed.

I have to admit that this market shift to bring broadband to MDUs caught me a bit by surprise. Many years ago Verizon showed that there is a successful business plan for building fiber to older residential neighborhoods. In the northeast Verizon still carries significant market share in its FiOS neighborhoods, and customers consistently rate them as having better customer service than the cable companies. Other telcos like CenturyLink are copying the Verizon model and are building swaths of fiber in residential neighborhoods.

The traditional wisdom was that it is too costly to bring fast broadband to apartments. A decade ago bringing fiber to an apartment meant rewiring the whole building with fiber – and for many apartments that is prohibitively expensive. But there have been technology advances that have made this more feasible. For example, much of the ‘near-gigabit’ speeds can be achieved by using G.Fast technology over existing coaxial or telco cable in older apartments. There have also been big improvements for indoor fiber deployments that include small flexible fibers and techniques for installing fiber inconspicuously in hallways. Many buildings that seemed too costly to serve years ago now make economic sense. Finally, the potential to deliver backhaul to an MDU using millimeter wave radios is going to eliminate the need to build as much fiber.

The real big unknown is how successful any of these big companies will be with 5G. As I’ve been writing lately there are still a lot of barriers that might make it difficult for AT&T to use the wireless technology to cover 10 million passings. We’re going to have to wait to see some real deployments over the next few years to see if the technology works as promised and if the cost of deployment is as cheap as anticipated. But the one thing that these analysts have gotten right is that the big telcos are finally fighting back against the cable monopolies they helped to create by sticking with DSL too long. It’s going to be interesting to see how well they do in winning back customers that they lost over the last two decades.

Fiber and the New Economy

Many communities look at having a fiber network as a catalyst for economic development. When fiber was relatively new this was clearly the case. We can look at some of the early adapters of fiber, like Bristol Virginia to see how communities leveraged a fiber network to bring large employers and jobs to their communities.

I’ve been talking to some economic development pros recently and I think that historic economic development model is rapidly changing due to the nature of our new economy. I don’t think there will ever come a time when communities won’t hope for a major employer to move to their area, and we know today that requires having great broadband. But I also think that smart communities will look beyond this model and will find new ways to best leverage fiber.

Probably the biggest recent change in our economy is the work-at-home phenomenon. People are able to work at home for major corporations, and that means they can live anywhere. But the real juice from the work-at-home economy is from people starting their own businesses from their homes.

I’ve worked out of my house for nearly twenty years. When I first started doing this it was rare and I didn’t know anybody else who worked full-time from home. But today its common – just on my one city block there are a half a dozen families supported by work-at-home jobs.

The other big economic trend is that we are becoming more of a service economy. There are numerous businesses making a go of it by supporting others in their community. A few big retail companies like Walmart and Amazon devastated a lot of the small retail community across the country. But I look around and see a thriving service community that is immune from the effects of big retail – restaurants, brew pubs, pet sitters, investment advisors, Lyft drivers, etc.

The new economy also fosters craftsmen and artists. Where I live it’s hard to find a carpenter or electrician since they are all so busy. As I travel around the country I see art and photography studios everywhere.

So what does all of this have to do with fiber and economic development? First, I think it’s becoming clear that communities without fiber are in danger of becoming irrelevant and of withering. The people working in the new economy need broadband to work at home or to sell their services or art work. And communities clearly need good broadband to support good education.

The new model I see for economic development recognizes the new economy and values it in the same way that they used to value drawing in the big employer. A person making $65,000 per year from their home is probably more valuable to a community than somebody making the same in a new factory that relocated for tax incentives. The factory worker might be commuting and carrying the salary elsewhere while the home worker is likely to shop and spend their money locally. And the factory is likely to be shipping profits out of town. We’ve known for decades that there is a huge multiplier effect from spending money locally, and creating jobs that spend locally can really fuels a local economy.

So I think the smart communities are those that will embrace this new economy and foster it. I would venture to say that very few communities even know how many local people work from their homes. And few have any strategy for attracting more people to work from home or for helping local people start new home-based businesses.

Since work-at-home employees can generally live anywhere, cities need a strategy if they want to foster this new economy. One possible strategy is to do everything possible to make a community into a place that people want to live. That means expending economic development resources to promote new restaurants, to foster the local art community, to create better parks and green spaces. It means cleaning up old downtowns and doing everything possible to get businesses to occupy empty spaces. It means eliminating tax or other disincentives for new small businesses. And it means making sure there is good enough broadband to support all of these efforts.

Cable Labs Analysis of 5G

Cable Labs and Arris just released an interesting paper that is the best independent look at the potential for 5G that I’ve seen. Titled ”Can a Fixed Wireless Last 100m Connection Really Compete with a Wired Connection and Will 5G Really Enable this Opportunity?”, the paper was written to inform cable companies about the potential for 5G as a direct competitor to cable network broadband. The paper was released at the recent SCTE-ISBE forum in Denver. The paper is heavily technical and is aimed at engineers who want to understand wireless performance.

As is typical with everything I’ve seen out of Cable Labs over the years the paper is not biased and takes a fair look at the issues. It’s basically an examination of how spectrum works in the real world. This is refreshing since the vast majority of materials available about 5G are sponsored by wireless vendors or the big wireless providers that have a vested interest in that market succeeding. I’ve found many of the claims about 5G to be over-exaggerated and optimistic in terms of the speeds that can be delivered and about when 5G will be commercially deployed.

The paper explores a number of different issues. It looks at wireless performance in a number of different frequency bands from 3.5 GHz through the millimeter save spectrum. It takes a fair look at interference issues, such as how foliage from different kinds of trees affects wireless performance. It considers line-of-sight versus near line-of-sight capabilities of radios.

The conclusions from the report are nearly the same ones I have been blogging about for a while:

  • Speeds on 5G can be significant, particularly with millimeter wave radios. The radios already in use today are capable of gigabit speeds.
  • The spectrums being used suffer significant interference issues. The spectrums will be hampered when being used in wooded areas or with the trees on many residential streets.
  • Coverage is also an issue since the effective delivery distance for much of the spectrum being used is relatively short. The means that transmitters need to be relatively close to customers.
  • Backhaul is a problem. Fast speeds require fiber connectivity to transmitters or else robust wireless backhaul – which suffers from the same coverage and interference issues as the connections to homes.

The paper also takes a look at the relative cost today of deploying 5G technology at today’s costs:

  • The CAPEX for a 3.5 GHz system used for wireless drops (800-meter coverage distance) costs $3,000 for the transmitter and $300 per home. These radios would be making home connections of perhaps 100 Mbps.
  • A millimeter wave transmitter costs about $22,500 with home receivers at about $650. This would only cover about a 200-meter distance.
  • In both cases the transmitter costs would be spread over the number of customers within the relatively short coverage area.
  • These numbers don’t include backhaul costs or the cost of somehow mounting the radios on poles in neighborhoods.
  • These numbers don’t add up to compelling case for 5G wireless as strong cable competitor, particularly considering the interference and other impediments.

The conclusion of the paper is that 5G will be most successful for now in niche applications. It is likely to be used most heavily in serving multi-tenant buildings in densely populated urban areas. It can be justified as a temporary solution for a broadband customer until a carrier can bring them fiber. And of course, we already know that point-to-multipoint wireless already has a big application in rural areas where there are no broadband alternatives – but that application is not 5G.

But for now, Cable Labs is telling its cable company owners that there doesn’t seem to be a viable business case for 5G as a solution for widespread deployment to residential homes in cities and suburbs where the cable companies operate.

The Rush to vMVPDs

To those of you not familiar with the industry lingo, a vMVPD is a virtual multichannel video programming distributor, or virtual cable company. This term is being used to describe OTT providers that offer a version of the same channels offered by cable companies. This sector includes Sling TV, DirecTV Now, Playstation Vue, Hulu Live, YouTube TV and a few others. These providers stream networks on the same linear schedule as is shown on cable TV. Providers of alternate programming like Netflix or Amazon Prime are not considered as vMVPDs.

Industry analysts say that the vMVPDs as a group gained over 900,000 customers in the recently ended third quarter. That is a startling number and represents almost one percent of the whole traditional cable TV market, all captured in just one quarter. We’ll have to wait a bit to see how the whole cable market performed. But we already know that Comcast lost over 150,000 cable customers for the quarter. Since they had been hanging onto cable customers better than the other cable companies I think we can expect a bloodbath.

This kind of explosive growth is perhaps the best harbinger for the slow death knell for traditional cable TV. This new industry is still less than three years old with Sling TV having launched in February 2015. The industry started slowly and had only a few hundred thousand customers at most by the end of 2015.

But it’s now obvious that a lot of people are deciding that they don’t want to pay the big monthly bill for the giant channel line-up. The analysis from Nielsen shows that most households only watch a handful of channels. While no vMVPD is probably going to give households exactly the channels they most want to watch, they are obviously providing enough channel choices to lure people away from the cable companies.

It’s an interesting transition to watch. To some degree the programmers are contributing to their own demise. When people leave a cable line-up of 200 channels to instead watch an vMVPD line-up of less than 50 channels there are obviously a lot of networks that are no longer collecting customer fees. Practically every network is bleeding customers and this shift to OTT viewing is going to kill off a lot of network channels. I read an interview a few months ago with the head of programming at Fox who believed that his company would shut down the majority of their cable networks within a few years.

Another thing I find interesting about this shift is that the vMVPDs are not particularly easy to use. I’ve now tried four of them – Sling TV, DirecTV Now, Playstation Vue and Fubo TV, and I will get around to trying them all eventually. None of them have the ease of use of a cable settop box. You can’t just surf through channels easily to see what’s on and you have to instead navigate through menus that take several steps compared to a simple ‘channel up’ command on a cable remote.

These four services also have channel guides of a sort, but they are also cumbersome to use. I’ve found that it can easily take three or four minutes to change between two shows, and that’s when you know what you want to watch. The guides on these services are not yet friendly for looking hours or days ahead to see what you might want to watch later. And at least one of the services, Playstation Vue, is so confusing that I often get lost in its menus.

And yet nearly a million people changed to one of these services in the last quarter. The biggest appeal for these services is price along with a total ease to subscribe or unsubscribe. After years of dealing with big cable companies I was apprehensive the first time I tried to unsubscribe to Sling TV – but it took less than a minute to do on-line and was not a hassle. The services differ in features like the number of people who can watch different programming at the same time on an account, but they are all becoming more people friendly over time.

At this point AT&T might be the only company that is getting this right. The company lost 385,000 customers in the third quarter between DirecTV satellite service and U-verse. But they gained 296,000 DirecTV Now customers to make up for a lot of those losses. At this point nobody is talking about the margins on vMVPD service, but it can’t be a whole lot worse than the shrinking margins on traditional cable TV.

I believe we are seeing the future of TV in the vMVPD product. We’ll probably look back five years from now and laugh at these hard-to-use first generation services. I’m sure that over time they will get far easier to use and I’m getting ready to experiment using my Amazon Echo to navigate through Playstation Vue. When it becomes simple to use vMVPDs, then  traditional cable TV might have become passe.

A New Vision of Economic Development

 

Photo by Drew C. Wilson of the Wilson Times


I attended a forum in Wilson, North Carolina last week that talked about how fiber is transforming their city. They talked about how they are trying a new model for economic development.

The traditional economic development model concentrated on searching for big piles of jobs. Communities made efforts to attract major employers and worked hard to keep companies from leaving their town. But it’s pretty obvious when looking around rural America that this model stopped working somewhere along the line. I visit far too many communities that have lost big employers and that are not finding anybody to replace them. This is due to some degree to the overall huge decrease in US manufacturing jobs. But it also is due in part to the general decline of businesses located in smaller communities.

Wilson is a community of around 50,000. Historically the city was known as the ‘world’s greatest tobacco market’ in the 19th century and tobacco was huge in the area until a few decades ago. Wilson was also the birthplace to BB&T bank, which is still the largest employer in the city. But like happened with many US cities, Wilson also went through a decline. Some small manufacturers closed and the tobacco business died. In a scene that is familiar across the country the downtown business district dried-up as retail moved to other places.

Wilson started its fiber optic business in 2008 under the tradename of Greenlight. They were one of the first cities in the country to offer gigabit broadband to residents. And that fiber network was the linchpin for the city in developing their new vision of economic development.

The concept behind Wilson’s vision sounds simple. They figure that that the best way to attract jobs to the community is by working to make their community a place where people want to live. They want visitors to the city to like it enough that some of them will want to move there. And they figure that when they reach that goal that businesses will naturally want to locate there. So they are looking to grow their economy by concentrating on and improving the assets they already have.

Of course, this is anything but simple. Many cities have tried this and only a few have found a way to rebound from the decaying downtowns we see all over the country. Wilson is making the turn by concentrating on the downtown area. They lured the Wilson Times, a local daily newspaper, to refurbish an old building and move back into downtown. They raised the money to renovate an old theater to create the Edna Boykin Cultural Center. There is a project to build new housing downtown next to the whirligig park (the picture accompanying this blog). They attracted Peak Demand to make a $2.6 M investment to manufacture electrical components in an old tobacco processing plant. And these investments are bringing back other businesses. There are new restaurants and two brew pubs that have opened in the downtown.

Wilson is using an approach that other cities should consider. They involve all of the stakeholders in the community in the effort to improve quality of life there. That includes working with Barton College, a 1,200-student liberal arts university and nursing school. They challenged the arts community to move and grow downtown and have a thriving art scene. They put an emphasis on buying local, which we all know has a tremendous local economic multiplier effect. The various constituencies in the city meet often to discuss ways to make the city better.

But they credit the fiber network for being the change that started everything. While big companies and big employers are important to every community, Wilson understood that the work-from-home entrepreneur movement is creating a lot of jobs and a lot of wealth. And so they foster innovation in a number of different ways and strive to make small and new businesses successful.

The big shame is that the North Carolina legislature passed a law to prohibit other communities in the state from following the Wilson model. Cities are no longer allowed to become retail ISPs in North Carolina. If they build fiber it has to be operated by somebody else – and we know that is a far harder model to make work. One only has to look at what’s happening in Wilson to understand that fiber is an important component these days for economic vitality. But fiber alone is not a guarantee for economic success. It takes a community-wide effort like the one in Wilson to take advantage of what fiber offers. Wilson still has a way to go, but you can feel the excitement in the community – and that is what makes any city a place where people want to live.

5G Networks and Neighborhoods

With all of the talk about the coming 5G technology revolution I thought it might be worth taking a little time to talk about what a 5G network means for the aesthetics of neighborhoods. Just what might a street getting 5G see in new construction that is not there today?

I live in Asheville, NC and our town is hilly and has a lot of trees. Trees are a major fixture in lots of towns in America, and people plant shade trees along streets and in yards even in states where there are not many trees outside of towns.

5G is being touted as a fiber replacement, capable of delivering speeds up to a gigabit to homes and businesses. This kind of 5G (which is different than 5G cellular) is going to use the millimeter wave spectrum bands. There are a few characteristics of that spectrum that defines how a 5G network must be deployed. This spectrum has extremely short wavelengths, and that means two things. First, the signal isn’t going to travel very far before the signal dissipates and grows too weak to deliver fast data. Second, these short wavelengths don’t penetrate anything. They won’t go through leaves, walls, or even through a person walking past the transmitter – so these frequencies require a true unimpeded line-of-sight connection.

These requirements are going to be problematic on the typical residential street. Go outside your own house and see if there is a perfect line-of-sight from any one pole to your home as well as to three or four of your neighbors. The required unimpeded path means there can be no tree, shrub or other impediment between the transmitter on a pole and each home getting this service. This may not be an issue in places with few trees like Phoenix, but it sure doesn’t look very feasible on my street. On my street the only way to make this work would be by imposing a severe tree trimming regime – something that I know most people in Asheville would resist. I would never buy this service if it meant butchering my old ornamental crepe myrtle. And tree trimming must then be maintained into the future to keep new growth from blocking signal paths.

Even where this can work, this is going to mean putting up some kind of small dish on each customer location in a place that has line-of-sight to the pole transmitter. This dish can’t go just anywhere on a house in the way that satellite TV dishes can often be put in places that aren’t very noticeable. While these dishes will be small, they must go where the transmitter can always see them. That’s going to create all sorts of problems if this is not the place in the home where the existing wiring comes into the home. In my home the wiring comes into the basement in the back of the house while the best line-of-sight options are in the front – and that is going to mean some costly new wiring by an ISP, which might negate the cost advantage of the 5G.

The next consideration is back-haul – how to get the broadband signals into and out of the neighborhood. Ideally this would be done with fiber. But I can’t see somebody spending the money to string fiber in a town like Asheville, or in most residential neighborhoods just to support wireless. The high cost of stringing fiber is the primary impediment today for getting a newer network into cities.

One of the primary alternatives to stringing fiber is to feed neighborhood 5G nodes with point-to-point microwave radio shots. In a neighborhood like mine these won’t be any more practical that the 5G signal paths. The solution I see being used for this kind of back-haul is to erect tall poles of 100’ to 120’ to provide a signal path over the tops of trees. I don’t think many neighborhoods are going to want to see a network of tall poles built around them. And tall poles still suffer the same line-of-sight issues. They still have to somehow beam the signal down to the 5G transmitters – and that means a lot more tree trimming.

All of this sounds dreadful enough, but to top it off the network I’ve described would be needed for a single wireless provider. If more than one company wants to provide wireless broadband then the number of devices multiply accordingly. The whole promise of 5G is that it will allow for multiple new competitors, and that implies a town filled with multiple wireless devices on poles.

And with all of these physical deployment issues there is still the cost issue. I haven’t seen any numbers for the cost of the needed neighborhood transmitters that makes a compelling business case for 5G.

I’m the first one to say that I’ll never declare that something can’t work because over time engineers might find solutions for some of these issues. But where the technology sits today this technology is not going to work on the typical residential street that is full of shade trees and relatively short poles. And that means that much of the talk about gigabit 5G is hype – nobody is going to be building a 5G network in my neighborhood, for the same sorts of reasons they aren’t building fiber here.

Big ISPs and Elections

Before you stop reading, this blog isn’t about party politics – the elections I am talking about are those where citizens vote on building a fiber optic network in their community. The incumbents don’t seem able to pass up the chance to turn an election their way when competition is put onto the ballot.

The latest example of this is the upcoming election on November 7 in Ft. Collins, Colorado. Voters in that community will be voting on whether to amend the city charter to allow the city to build and operate a fiber optic network in the city. Colorado law makes this elections mandatory, but I’ve seen other cities hold voluntary elections on the issue so that they are certain that the citizens are behind their efforts to build fiber. A positive vote in Ft. Collins would allow the city to take the next step to investigate if they want to build a fiber network in the city.

Ft. Collins is a community of 59,000 homes and Comcast and the other incumbent ISPs have spent over $200,000 so far in advertising against the ballot measure – a phenomenal amount of money spent on a local election and the most ever seen in Ft. Collins.

As is usual for fiber ballot initiatives, the incumbents are fighting against the passage of the measure by spreading lies and misinformation. For example, in Ft. Collins they are saying that voting for the measure would preclude the city from making other infrastructure upgrades for things like roads. In fact, this ballot measure just gives the city the legal authority to explore fiber and it’s likely that they would have another election to approve a bond measure if they decide to float a bond for fiber – a decision that would be some time in the future.

The misinformation being floated in Ft. Collins is tame compared to some of the other ways that incumbents have tried to stop fiber initiatives. In Lafayette Louisiana the combination of Cox and BellSouth (now AT&T) were extremely aggressive in trying to stop the fiber initiative (including filing several lawsuits to stop the effort). But prior to the election when fiber was going to be on the ballot they called every home in the community with a push poll that asked ludicrous questions about the fiber project. An alert citizen recorded the push poll and it can be found here. This takes 30 minutes to hear the whole thing, but if you are interested in the tactics the big ISPs use to fight it, this is well worth a listen. There are some amazing questions in this poll, and the gall of this push poll might have been what pushed the election to pre-fiber. In Louisiana the city needed to get more than a 65% yes on the fiber initiative, and due to a strong community effort the ballot measure passed easily.

I also remember a similar election in North St. Paul, Minnesota, a small community surrounded by the city of St. Paul. When the city put a fiber initiative on the ballot Comcast sent busloads of people to the city who went door-to-door to talk people out of voting for fiber. They deployed the usual misinformation campaign and scared a community that had a lot of elderly citizens into voting against the fiber initiative, which narrowly lost at the polls.

There was a similar lection recently in Longmont, Colorado. When the city first held a vote on the same ballot measure as Ft. Collins, the money from the big ISPs defeated the ballot measure. The ISPs won using a misinformation campaign that talked about how the fiber effort would raise taxes. But the citizens there really wanted fiber, and so they asked for a second vote and in the second election there was a massive grass-roots effort to inform the community about the facts. The fiber initiative on the second ballot won resoundingly and the city now has its fiber network.

There are several lessons to be learned from these ballot battles. First, the incumbents are willing to make a sizable investment to stop competition. But what they are spending, like the $200,000 in Ft. Collins, is a drop in the bucket compared to what they stand to lose. Second, they always attack fiber initiatives with misinformation, such as scaring people about higher taxes. They don’t fight by telling what a good job they are doing with broadband And finally, we’ve seen the ISP efforts be successful unless there is a strong grass-roots effort to battle against their lies. Cities are not allowed by law to take sides in ballot initiatives during an election cycle and must sit quietly on the sidelines. And so it’s up to citizens to take on the incumbents if they want fiber. The big ISPs will always outspend the pro-fiber side, but we’ve seen organized grass-roots efforts beat the big money almost every time.

FCC Wants to Change 3.5 GHz Spectrum Rules

The FCC voted last week to re-examine the rules for the deployment of 3.5 GHz spectrum for wireless broadband. This is the spectrum that has generally been referred to as Citizen’s Band Radio. This change clearly favors large carriers over the small carriers which were the targeted users from the existing rules.

The specific changes proposed by the rules include:

  • Lengthened the length of a license from 1 year to 10 years.
  • Eliminate the rules that the exclusivity of a license expires at the end of the first license term. Exclusivity can now extend into a license renewal.
  • Increase the size of the geographic footprint of a license. The license area before was a census tract, which is generally an area encompassing 2,500 to 8,000 people. The Census views a tract as the equivalent of a ‘neighborhood’. The new licenses areas are proposed to be something larger like entire counties or else Partial Economic Areas (PEAs). PEAs were defined in the recent incentive auctions and subdivide the country into 416 PEA regions.
  • Allows license holders to partition and disaggregate licenses between adjacent geographic areas.
  • Eliminated the rules that limited the number of licenses that can be held by one entity in an area. This also would allow license holders to bid on the use of individual channels.

What does all of this mean? This is largely a shift to allow big wireless carriers to obtain and use the spectrum for cellular service. Before the spectrum rules were aimed at benefiting small rural broadband providers. They would have been able to get a license for a small geographic area and they then got a 1-year head-start to deploy the spectrum before anybody else. The first licensee then had an advantage because future deployments had to be synchronized to not interfere with them.

The old rules made it difficult, but not impossible, for the bigger companies to use the spectrum. A cellular provider was not likely to invest in small license footprints and only be protected for a year from competition and interference. But the new rules allow for a much bigger footprint, similar to that used for other cellular spectrum. And the ten-year license provides a long-term opportunity for no competition, as well as a chance to renew the original license.

Basically this is a spectrum grab by the cellular providers to use for LTE or 5G cellular. Two of the big proponents of these changes include Comcast and Charter which want their own spectrum to support their new cellular businesses.

This change will make it much harder for rural deployments by WISPs and other ISPs willing to serve customers with wireless connections. The original rules also envisioned that this spectrum would enable smaller carriers to deploy various small-cell technologies and not just point-to-multipoint radios.

This is another proposed ruling that shows that current FCC is now clearly pro-big business. Almost every ruling they’ve made so far benefits big companies – the big ISPs, the big TV station owners, and the big wireless carriers. This particular ruling is a big give-away to the cellular companies and to Comcast and Charter. Under the rules the spectrum can be licensed inexpensively compared to spectrum that is auctioned. The new rules allowing large coverage areas will greatly disadvantage small carriers that only want to license a small service area – which was the entire purpose of the original rules for the spectrum.

The FCC voted 4-1 to consider the new rules, which is a likely indication that the new rules will be adopted after the required deliberation time required by FCC rules.

Broadband Speeds are a Local Issue

You might think that the big ISPs deliver the same broadband products everywhere. But I’ve been seeing evidence that broadband speeds are definitely a local issue. One of the products that we’ve been using to help clients assess a new market is to get a lot of people in the potential market to take speed test. We’ve mostly been using the Ookla speed test, but probably any speed test is sufficient as long as everybody in a market takes the same test.

The results of these speed tests surprised me a bit because they showed a wide variance in the products of the major ISPs. For example, I’ve seen markets where Comcast is delivering a little more download speed than they are advertising. But I also saw tests results from a Comcast market where the speeds were about 20% less than advertised. I’ve seen the same thing with AT&T where there are markets that get only half of the advertised speeds and other markets where they were mostly delivering what they are promising. I’m not sure if there is any better demonstration that speeds are a local issue than by seeing that the big ISPs don’t deliver the same speeds in every market.

There is a long list of reasons that can account for the differences in speeds. A big one is the age and quality of the network cables. Older telco copper and older coaxial cables can cause a lot of problems with quality. The size of customer nodes is always an issue. If everything else is equal, a cable company node serving 100 customers is going to have better broadband speeds than one serving 200 customers.

The other big issue that affects customer performance is what I call network choke points. A chokepoint is any place in a broadband network that restricts the flow of data to and from customers. There can be a choke point directly within a neighborhood if the nodes are too large. There can be a chokepoint between a node and the core network if the electronics for the connection are undersized. There can be a chokepoint on local network rings if they don’t provide enough bandwidth. There can be electronics chokepoints at a headend if a router or other major piece of electronics is overwhelmed. And finally, there can be an overall chokepoint in a network if the data pipe going to the Internet is too small.

Chokepoints don’t have to always be a problem. Many chokepoints only appear during the busiest hours of usage on the network, but don’t impede data speeds when data traffic volumes are smaller. And this means that chokepoints are often hyper-local. They might affect one neighborhood but not the one next door, and only at some times of the day. I’m guessing that the slowest results I saw in the big ISP speed tests were during the peak evening hours.

These chokepoints obviously don’t only affect the large ISPs and plenty of smaller ISP networks have chokepoints. I’ve seen numerous network chokepoints appear in recent years due to the explosive growth of the use of broadband. A network that may have been functioning perfectly a few years ago will develop chokepoints as the amount of total bandwidth on networks overwhelm some portion of a network.

ISPs often are challenged to keep up with the upgrades needed to avoid chokepoints, because generally the only ways to relieve chokepoints is to replace cables or to upgrade electronics, which can be expensive. Smaller ISPs often don’t have the immediate capital available to fix chokepoints as they appear. The big ISPs tend to ignore chokepoints as they appear and to make large fork-lift upgrades periodically instead of making the constant small upgrades needed to keep the network working perfectly.

I always advice my clients to keep a running list of all of their chokepoints. With good network engineering and monitoring practices a company can see chokepoints coming long before they materialize and hopefully can plan to make the needed upgrades before they degrade the customer experience.

 

FCC’s Recommendations to Avoid Network Outages

The FCC’s Public Safety and Homeland Security Bureau just released a list of recommended network practices. These recommendations are not a comprehensive list of good network practices, but rather are compiled by analyzing the actual network outages reported to the FCC over the last five years. Telcos are required to notify the FCC of significant network outages and every item on this list represents multiple actual network outages. It’s easy to look at some of the items on the list as think they are common sense, but there obviously there are regulated telcos that triggered had outages due to ignoring each of these network practices.

Following are some of the more interesting recommendations on the list:

Network Operators, Service Providers and Property Managers together with the Power Company and other tenants in the location, should verify that aerial power lines are not in conflict with hazards that could produce a loss of service during high winds or icy conditions. This speaks to having a regular inspection and tree trimming process to minimize damage from bad storms.

Network Operators and Property Managers should consider pre-arranging contact information and access to restoral information with local power companies. This seems like common sense, but I’ve been involved in outages where the technicians did not know how to immediately contact other utilities.

Network Operators, Service Providers and Public Safety should establish a routing plan so that in the case of lost connectivity or disaster impact affecting a Public Safety Answering Point (PSAP), 9-1-1 calls are routed to an alternate PSAP answering point. A lot of the recommendations on the FCC’s list involve 9-1-1 and involve having contingency plans in place to keep 9-1-1 working in the case of network failures.

Network Operators, Public Safety, and Property Managers should consider conducting physical site audits after a major event (e.g., weather, earthquake, auto wreck) to ensure the physical integrity and orientation of hardware has not been compromised. It’s easy to assume that sites that look undamaged after big storms are okay. But damage often doesn’t manifest as outages until days, weeks or months later.

Network Operators and Service Providers should verify both local and remote alarms and remote network element maintenance access on all new critical equipment installed in the network, before it is placed into service. I’ve seen outages where equipment was installed but the alarms were not tested. You don’t want to find out that an alarm isn’t working when it’s needed.

Network Operators, Service Providers, Public Safety and Property Managers should engage in preventative maintenance programs for network site support systems including emergency power generators, UPS, DC plant (including batteries), HVAC units, and fire suppression systems. This might easily be the biggest cause of network outages. ISPs get busy and don’t test all of the components critical to maintaining systems. A lot of outages I’ve been involved with were due to failures of minor components like fans or air conditioning compressors.

Network Operators, Service Providers, Public Safety, and Equipment Suppliers should consider the development of a vital records program to protect vital records that may be critical to restoration efforts. Today there is often software, databases and other vital records that must be restored in order first to get equipment up and functioning. Electronics records of this type need to be kept in a secure system that is separate and doesn’t rely on the network to be functioning, but that also can be accessed easily when needed.

Network Operators, Service Providers, Public Safety and Property Managers should take appropriate precautions to ensure that fuel supplies and alternate sources of power are available for critical installations in the event of major disruptions in a geographic area (e.g., hurricane, earthquake, pipeline disruption). Consider contingency contracts in advance with clear terms and conditions (e.g., Delivery time commitments, T&Cs). This is a lesson most recently experienced after the recent hurricanes where local gasoline supplies dried up and several utilities without their own private fuel supply were stranded along with the rest of the public.

This FCC list is a great reminder that it’s always a good idea to periodically assess your disaster and outage readiness. You don’t want to discover gaps in your processes during the middle of an outage.