Monthly Archives: March 2021

Automation and Fiber

We have clearly entered the age of robots, which can be witnessed in new factories where robots excel at repetitive tasks that require precision. I read an interesting blog at Telescent that talks about using robots to perform routine tasks inside large data centers. Modern data centers are mostly rooms full of huge numbers of switches and routers, and those devices require numerous fiber connections.

The blog talks about the solvable challenges of automating the process of performing huge volumes of fiber cross-connects in data centers. Doing cross-connects with robots would allow for fiber connections to be made 24/7 as needed while improving accuracy. Anybody who has ever been in a big data center can appreciate the challenge of negotiating the maze of fibers running between devices. The Telescent blog predicts that we’ll be seeing the accelerated use of robots in data centers over the next few years as robot technology improves.

This raises the interesting question of whether we’ll ever see robots in fiber networks. As an industry, we’ve already done a good job of automating the most repetitive tasks in our telco, cable, and cellular central offices. Most carriers have automated functions like activating new customers, changing products and features, and disconnecting customers. This has been accomplished through software, and the savings for automation software are significant, as described in this article from Cisco.

But is there a future in the telecom industry for physical robot automation? I look around the industry and the most labor-intensive and repetitive processes are done while building new networks. There probably is no more meticulous and repetitive task than splicing fibers during the construction process or when fixing damaged fibers. Splicing fiber is almost the same process used in the past to splice large telephone copper cables. A technician must match the same fiber from both sheathes to create the needed end-to-end connection in the fiber. This isn’t too hard to do when splicing a 12-fiber cable but is challenging when splicing 144 or 288-count fibers in outdoor conditions. This is even more challenging when making emergency repairs on aerial fiber in a rain or snowstorm in the dark.

This is the kind of task that robots could master and perform perfectly. It’s not hard to imagine feeding both ends of fiber into a robotized box and then just waiting for the robot to make all of the needed connections and splices perfectly, regardless of the time of day or weather conditions.

I had a recent blog that talked about the shortage of experienced telecom technicians, and splicers are already one of the hardest technicians for construction companies to find. As we keep expanding fiber construction, we’re liable to find projects that get bogged down due to a lack of splicers.

I have no idea if any robot company has even thought about automating the splicing function. We are in the infancy of introducing robots into the workplace and there are hundreds of other repetitive tasks that are likely to be automated before fiber splicing. There might be other functions in the industry that can also be automated if robots get smart enough. The whole industry would emit a huge sigh of relief if robots could tackle make-ready work on poles.

Quantifying the Benefits of Fiber

Dr. Bento J. Lobo, an economist at the University of Tennessee at Chattanooga undertook a study to quantify the benefits of the municipally-owned fiber network in Chattanooga. Any citywide fiber network brings economic development to a community, but a municipally-owned system brings additional benefits because of the way that the business is more deeply integrated into the community.

The study estimates that the network has generated nearly $2.7 billion in benefits since the network was constructed a decade ago. I’ve always felt that you have to take the claims from economic benefit studies with a grain of salt. Many of the benefits are easily measurable, but other benefits rely on assumptions that are hard to prove or disprove. But the big story is that even a conservative economic analysis (and this analysis might already be conservative) would still demonstrate a huge benefit to the city from fiber.

Chattanooga is a bit of a unique case because it was one of the first municipally owned citywide fiber systems and is also the largest. Being first to market with fiber brought some benefits to Chattanooga that might not come in the same magnitude to other cities building fiber today. For example, EPB – the municipal utility, offered affordable gigabit broadband when the cable company still had speeds of 30 Mbps download.  With that said, here are some of the benefits that the fiber business brought to Chattanooga.

  • One of the most immediate benefits is a sizable saving to consumers. EPB charges $57.99 for a symmetrical 300 Mbps connection and $67.99 for a symmetrical gigabit. EPB charges $9.99 for a whole-house WiFi network. These prices are far lower than Comcast’s basic product in most markets that is $76 for up to 200/25 Mbps connection plus $14.95 for the WiFi modem. EPB also offers low-income broadband for qualifying homes at $26.99 per month. The study quantifies the benefit to consumers of over $144 million.
  • Chattanooga was one of the first places in the nation that touted the ability to bring an affordable gigabit to anybody. This resulted in attracting entrepreneurs to the city that were enticed by cheap and fast broadband. The study estimates that the fiber network has gained or saved over 9,500 jobs. The city has gained a reputation as a good incubator for start-ups and now has an innovative business community – where nothing like it existed before the network was built. Cities building fiber today are not going to duplicate this results, at least not to the extent seen in Chattanooga.
  • The study also credits the fiber network with attracting nearly $1 billion in new investments by businesses in the city – from start-up or existing corporations that moved to the city or expanded existing businesses.
  • There are a few direct benefits that come from operating a new municipal business. For example, the fiber business has paid around $60 million into the city coffers for in-lieu-of-tax payments (payments that mimic taxes). The fiber business has also absorbed $338 million in overhead costs that would have otherwise been charged to the electric utility – a saving that translates into lower electric rates.
  • The study assigns a $750 million benefit to an electric smart grid. Fiber allowed a number of beneficial changes to the operations of the electric business. This includes automated meter reading that significantly lowered labor and vehicle costs. The fiber network has been credited with reducing the duration of electrical outages. The use of automated electrical switching devices has shaved the peak load (the biggest cost of power is to satisfy the peak during heavy usage times). Smart metering has saved the city over $50 million in electricity theft.
  • The study calculated $74 million in benefits to businesses for having faster and lower latency broadband due to automation and increased efficiency. This is the kind of benefit that is clearly real, but difficult to quantify.
  • The study also took a stab at quantifying more esoteric, but real benefits. For example, it’s incredibly hard to quantify the impact of having good broadband on education both before and during the pandemic. The city has done more than most cities in getting broadband to low-income and student households. There are also significant societal benefits from telecommuting, which was booming in Chattanooga even before the pandemic. Chattanooga was also pioneering telemedicine before the pandemic.
  • Finally, the study even assigns a value to the publicity generated by the fiber network. The city went from obscurity to appear on nationwide lists of the best places to live and retire, all credited to the fiber network.

Again, Chattanooga is somewhat unique by being an early adapter to fiber. But the city has also done a lot more than just offering affordable broadband. New businesses didn’t just magically appear in the city and are the result of the hard effort of economic development folks – but such efforts would have gone nowhere without the fiber network. Chattanooga is ahead of most cities in adopting the benefits of smart grid – but again, because it decided to take maximum of advantage of the fiber network.

My guess is that this study is a little conservative. For example, what’s the benefit from higher property tax revenues due to having Chattanooga listed as one of the best places in the country to live? I can’t even begin to imagine quantifying the true long-term benefits that come from using fiber to reduce the number of students that drop out of school or fall through the cracks. What’s the value to society from a student who goes to college, but without home broadband wouldn’t have?

No other city is likely to see some of Chattanooga’s early adopter advantages, but most of the advantages realized by the city can be duplicated. There is also a big lesson to be learned from Chattanooga – building fiber is only part of the story and a community has to take the extra steps to make sure that all of the constituencies of a city see the advantages of fiber. Too often, cities analyze the feasibility of a fiber network by looking only at the financial performance of the fiber business. Chattanooga’s example shows that the biggest advantages to the community don’t appear on the books of the fiber business, but in the pocketbooks of the citizens and businesses in the community.

Disappointment with the New FCC Mapping

The FCC took its latest shot at reforming the notoriously inaccurate broadband maps in January.  I put off writing a blog on this until now because I kept hoping that as I reread the new rules that I’d see something positive. But as I’ve reread the details of the proposed new mapping process, I see little improvement on the way. I’m not going to go through all of the details of the FCC order, just point out the proposed areas of the most impact.

The best change in reporting is to require ISPs to draw polygons around areas where customers either have service or where the ISP is willing to provide service within 10 days of a request. This will clean up two problems. It will draw lines around areas where cable company coverage stops in towns and create a clear border. Today, reporting by Census block often shows cable coverage extending far into the rural areas surrounding towns. Second, the consumer challenge process ought to eventually chop down over on rural WISPs and telcos that claim coverage where they can’t provide service.

The FCC totally blew the most important issue with poor mapping because it will continue to allow ISPs to report the fastest advertised broadband speed. This is the primary problem in rural areas today where the big telcos claim 25/3 Mbps advertised speeds and then deliver a 2 Mbps download speed. By not tackling the misrepresented speeds, the FCC really is wasting everybody’s time because this will produce a rural database that is no better than what we have today. Who cares what Frontier or CenturyLink advertise in rural America? We care about the poor speeds the telcos deliver.

The new rules include a two-tier challenge process, One form of challenge can come from governments or Tribes. The government challenge is complex in that anybody that wants to challenge must draw their own versions of the polygons in an area they are challenging. It will be difficult for governments to gather the huge volume of consumer data needed to make such a challenge. A government might gather a thousand speed tests in a rural county and still be unable to draw an accurate polygon of the coverage area. I foresee governments undertaking these challenges, but the process looks to be heavily slanted in favor of ISPs.

There is also a direct consumer challenge, but I think this public is going to be quickly disappointed by the process. A consumer can challenge that a broadband product is available at their home, and if they win, the carrier simply must redraw the polygon to exclude them – consumer challenges won’t bring anybody better broadband. Consumers are mostly going to want to challenge broadband speeds being delivered, but it’s highly unlikely that consumer challenges will succeed since ISPs are perfectly justified in reporting advertised speeds.

The FCC has a testing regimen that it says can be used to resolve major differences between the ISPs and the public – and they tout currently performing a few thousand speed tests per year. Does the FCC not realize that there are millions of homes that are misclassified in today’s mapping? The FCC would have to oversee millions of tests to respond to the flood of challenges that are going to be coming to the public under the new mapping system – and that’s not going to happen.

The proposed FCC mapping rules are not going to fix the problem that the public most cares about. By accepting advertised speeds today, the FCC has excluded huge areas with dreadfully poor broadband from being eligible for federal broadband grants. As long as the FCC defines grant areas by speed rather than by technology, and as long as the FCC keeps allowing ISPs to report advertised speeds, the FCC databases and maps will continue to ignore the reality of rural broadband and will continue to exclude areas from grant availability.

If the FCC moves forward with the recently adapted mapping rules it is headed for a disaster. The agency is foolish to establish a process to allow the public to challenge ISP reporting if it ends up ignoring the vast majority of those challenges. The agency is in for a public relations disaster of epic proportions. I seriously doubt that the FCC understands how irate the public is over poor treatment by ISPs in rural America – and a poorly managed challenge process is going to redirect that anger towards the FCC.

I can already imagine the response the public is going to get to an FCC challenge. “Dear Consumer: We’re sorry you are only getting 2 Mbps broadband service. Your ISP is properly reporting speeds in your area where it advertises a speed of ‘up to 25/3 Mbps’. We are not planning on funding any broadband grants in your area because we are happy with any ISP that advertises 25/3 Mbps broadband. Further, your ISP tells us that you did not properly perform the speed test, so we must reject your challenge. We hope this email is able to reach you. Have a nice day. Sincerely, the FCC”.

Why Do We Assume Cable Broadband is Always Good?

One of the oddest aspects of FCC monitoring of broadband is that the agency has accepted the premise that any broadband product faster than 25/3 Mbps is adequate broadband. This means that the FCC has completely accepted that broadband provided by cable companies is adequate and is something the agency doesn’t have to be concerned with. The FCC makes the automatic assumption that broadband from cable companies is good broadband and that anybody that has cable broadband is ‘served’ with broadband. The realities of the marketplace tell a different story, and cable broadband is often not adequate for homes and businesses that often have no other broadband option.

This became crystal clear during the pandemic when many homes found out that their cable broadband connection would not support the family for working and schooling from home. Families that were paying more than $70 per month for a broadband connection found that no more than one or two family members could work from home at the same time. This is all due to the decision of most cable companies to not upgrade the upload link when they implemented the upgrade to DOCSIS 3.1. Instead, cable companies stayed with the older DOCSIS 3.0 upload technology, and most homes get between 10 and 20 Mbps upload speeds, with some unlucky homes getting slower speeds. What no cable company ever talks about is the fact that upload bandwidth is jammed into the noisiest and part of the cable TV system that suffers from various kinds of interference that can further degrade performance.

We shouldn’t let the cable companies off the hook by blaming upload problems on the pandemic. Many cable company networks had big problems long before the pandemic. We recently did speed tests in a city where 75% of households in the city were seeing download speeds that were lower than the published target 100 Mbps target speed. 40% of the customers in that city were seeing speeds under 50 Mbps, with 15% seeing speeds under 25 Mbps. These kinds of speed test results demonstrate a network with problems. What are some of the problems that can lead to poor performance in a cable network?

  • Some cable companies still have network nodes that are too large, meaning too many households are sharing a fiber connection back to the node. It’s easy to spot such networks because they still bog down at prime time in the evenings when most households are using bandwidth. To give credit, most cable networks have licked this problem, but not all.
  • A common problem is poorly functioning cascading. Cable companies routinely have to amplify the signal between homes and the network core, and cascading problems occur when multiple amplifiers in a row serving a given neighborhood are not properly in sync, resulting in loss or distortion of data.
  • Everybody in the country talks about the poor condition of the telephone copper networks because of age. But the copper coaxial networks of the cable companies are also old and near the end of economic life. The majority of cable copper was built in the 1970s with some neighborhoods even earlier. Age is a problem with cable networks because a coaxial copper network acts like a giant antenna and picks up interference from numerous sources in the environment. New coaxial networks are ‘tight’, meaning that cable splice points don’t let in much interference. But over time, as cables are cut and repaired and as connections naturally loosen over time, the networks get noisier and noisier, meaning degradation of signal quality.
  • One of the most susceptible parts of the cable network are drops. They get whipped by the wind and degraded by weather.
  • There can also be tons of small network issues that never get fixed. There might be bad power taps or amplifiers that should be replaced. These devices might not have been properly installed and have been causing local problems for a few customers for decades.
  • There are documented examples of cable companies who have upgraded the network but didn’t upgrade modems. The most notorious such case was Charter in upstate New York where the company went for years with outdated modems, even after promising regulators the problem would be fixed. The company nearly lost its statewide franchise due to sticking with outdated customer technology.

A big part of my practice these days is working with cities that are fed up with inferior cable broadband. In some cities the strength of the broadband signal varies widely throughout the day. Some cities complain of nagging small outages that occur daily, with broadband dropping out of service for a few minutes or an hour or two at a time. As evidenced by the speed tests I’ve mentioned, some cities have broadband speeds that are far slower than what the cable company advertises. I’ve done several surveys this year where a third of homes say that cable broadband is not adequate for working or schooling.

Unfortunately, we have an FCC that assumes that everything faster than 25/3 Mbps is adequate broadband and there has been virtually no discussion at the FCC about cable companies that deliver lousy broadband. As household demand continues to grow, there is a growing number of customers who are unhappy with the broadband they are receiving from cable companies. City governments hear numerous complaints from the public about poor broadband but feel powerless in eliciting changes from cable company monopolies that could improve performance but refuse to do so. In most cases, the cable companies know exactly what their problems are in a given market but have elected to not spend the money to fix issues.