Monthly Archives: December 2015

Building Fiber to the X

Fiber CableAnybody that is a client of CCG probably knows that we build business plans as one of our primary products. Over the years we have built business plans for just about every kind of network and technology imaginable. And as you might suspect, these days we build a whole lot of fiber-to-the-premise business models for markets of all sizes.

I recently realized that we are on the verge of being able to use FTTP networks to do a lot more than serve residents and businesses. My typical business plan concentrates on the revenue streams that can be gotten from residential and business customers. We also look at a few other revenue streams like selling to large customers like schools or cell phone towers. And we normally consider wholesale sales made to other carriers already in the market.

But everything I read tells me that there are soon going to a whole lot of new opportunities for using a fiber network, particularly in medium to large markets. We are seeing some of these opportunities today, but each of these areas promises to get larger as time goes by. Consider some of the following:

Outdoor WiFi Network: More and more cities and even some carriers are starting to foresee business plans that include numerous outdoor WiFi hotspots. This can be for law enforcement and municipal use, for roaming WiFi cellphones, to provide a digital divide solution, or just to provide a service to citizens or customers.

Smart Lampposts: Similar to WiFi hotspots are smart lampposts that include smarter lighting that saves energy coupled with WiFi hotspots.

Mini Cell Sites: As the cellular carriers contemplate going to 5G they are going to need a lot more and smaller cell sites located close to users. And these cell sites will need fiber.

Traffic Light Systems: There are new systems of intelligent traffic signals that promise to significantly improve traffic flows using AI.

Cameras: There is a proliferation of cameras today including private security cameras, traffic cameras, web-cams, and law enforcement security monitoring.

Digital Advertising Signs: There are now programmable billboards that can change the display through programming.

IoT Aggregation Points: Most carriers envision a network of IoT aggregation points that gather the traffic to and from outdoor and other IoT monitors as a separate Ethernet network.

Smart Meter Aggregation Points: Although wireless technology seems to have won the smart meter race, these networks need neighborhood aggregation points to gather signals from the wireless monitors.

SCADA Systems for Electric and Water Utility Monitoring: While many electric and water systems now have fiber networks to connect monitors in their system, as we develop more sophisticated monitoring there will be the need for more monitoring points.

On top of these applications there are others that nobody has yet thought of, into a business plan. But fiber is a long-term investment and there are going to be numerous revenue opportunities like these that are going to help to pay for fiber, assuming that a fiber network is deployed with these kinds of connections in mind. People always ask me what happens to a fiber network as cable TV and telephone penetration rates drop, and at least part of the answer is that there are going to be numerous locations in a community that are going to require a fiber connection. None of us knows yet how these future revenue streams will be priced, but we know these are all revenue opportunities and that they are all coming in the not-too-distant future.

T-Mobile and Comcast Challenge Net Neutrality

Network_neutrality_poster_symbolBoth Comcast and T-Mobile have developed video plans that seem to be a direct challenge to network neutrality. In both cases they are playing favorites with video services in an environment where both carriers have data caps. I find it interesting that they would challenge the FCC in this manner while net neutrality is still being decided by the courts. Even should the FCC look at these cases, any decisions they make could be later changed by the court ruling. Carriers in general hate uncertainty and usually shy away from tackling regulatory matters that are under court review.

Comcast has come out with a skinny bundled they are calling Stream TV. It is being tested now in a few markets. The new service includes CBS, NBC, ABC, FOX, the CW, Univision, Telemundo, and HBO. It does not include the more popular cable networks like ESPN or AMC. The monthly price is $15, which is interesting since HBO NOW is sold at that same price if you buy it from HBO. It’s available only to Comcast broadband customers – it’s starting now in Boston and Chicago but should be available in the whole Comcast footprint by early 2016.

The service can be watched on any device in a home connected to the XFINITY network, but does not come to the Comcast settop box. A customer could watch it on a smart TV or by using Roku, Apple TV or similar devices. Comcast says this is a product aimed at cord cutters and at young viewers who don’t want to sit in front of a television. Everybody has been expecting these kinds of offerings from the big cable companies as a way to keep cord cutters sending them a monthly check.

There are two regulatory issues with this new offering. The first is that it does not fit into the scheme of the prescribed kinds of lineups defined by various cable laws enacted by Congress. It comes closest to being a basic cable line-up since it includes the network channels, but it excludes things like PBS and local government access channels. I think Comcast is trying to get around this by having this not delivered to the settop box. It’s instead delivered on the IP path. But there still seems to be room for a challenge to the legality of the offering because the various federal rules on cable tiers are silent about technology differences and the same sorts of line-up requirements apply to somebody like a Verizon FiOS network.

The net neutrality problem comes because Comcast is not going to count the bandwidth from the Stream TV service against their newly established 300 gigabit per month data caps, which are being trialed in a few markets but are expected everywhere. That seems to be a direct slap in the face to the FCC since net neutrality rules prohibit favoring your own products over those of competitors. It’s really hard to see how this can be allowed to stand. Watch Comcast without a data cap or some other alternate provider like Hulu and it counts against the cap.

The T-Mobile product takes a totally different tactic. T-Mobile has rolled out their own skinny bundle called Binge On. The product includes a wide range of channels like ESPN, HBO, Netflix, Showtime, and Hulu with a total of 23 choices. Customers must buy each of these separately, and the beauty of that is that customers get to put together their own skinny bundles of just what they want to watch.

The net neutrality issue is that none of the video viewing will count against a T-Mobile customer’s data cap. On quick glance that doesn’t look to be discriminatory since there is a wide range of possible programming to buy. But the rub comes in that if a customer watches other video on their phone from a site that is not part of the package then it counts against their cap.

This means that T-Mobile is picking winners and losers for video streaming. They are allowing a small handful of programming to be unmetered but exclude the other thousand sources of video. This exact situation was discussed by the FCC when they first proposed the new net neutrality rules and their worry was that allowing carriers to pick winners and losers would result in stifling new innovation on the web. If all of the carriers mimicked T-Mobile then a new video offering would have a huge uphill battle and we would have an oligopoly of a handful of the largest video providers.

Interestingly, FCC Chairman Tom Wheeler said that he didn’t see any problem with what T-Mobile is doing and that it sounded “highly innovative and highly competitive.” I don’t think that the video providers not included on the T-Mobile list agree with that, nor should they. It’s not hard to picture every large ISP making similar deals with the current OTT providers like Netflix, Hulu, and a few others and effectively shutting out any future new video providers from the majority of customers in the US.

Some Interesting Cable Statistics

television-sony-en-casa-de-mis-padresDigitalsmiths recently released their Q3 2015 Video Trend Report and there are some really interesting statistics to be gleaned from the report. This is a large survey given to 3,153 consumers in the US and Canada. I’d love to hear from any small service providers who thinks that the statistics for your own customers are much different than these.

Satisfaction with Current Provider: Only 53% of customers said they were happy with their current cable provider. 4.8% said they were going to cut cable service within the next six months, 7.2% said they were going to change providers, and 32% said they might change providers. We know from past surveys that many of the people who say they are going to drop cable don’t end up doing so, but these statistics show the general lack of satisfaction with whoever provides cable.

Size of Monthly Bill: This asked how much people spend on TV, Internet, and phone. 61% are spending more than $100 per month. 41% are spending more than $125 per month and 24% are spending more than $150. In 2013 56% of people spend more than $100.

Premium Programming: 24% of respondents buy HBO, 15% Showtime, 14% The Movie Channel, 10% Cinemax, and 10% Starz!. 12% of households buy a premium sports package.

Growing Awareness of Skinny Bundles: The survey defined skinny bundles as Hulu, HBO Now, Sling TV, CBS all Access, and the online Showtime. 63% are aware of these services, up from 56% in the first quarter of 2015.

Most Wanted for a la Carte: People were asked what channels they would most want to buy on an a la carte basis. Over 50% of the people would buy ABC, the Discovery Channel, CBS, NBC, the History Channel, and A&E. Over 40% would buy Fox, HBO, National Geographic, PBS, Comedy Central, and AMC. When asked how much people would be willing to spend in total for a la carte programming, the average was $40.50 with 22% not willing to pay more than $20 and only 4% willing to pay more than $81.

Feelings about Large Cable Packages: 34% of people are overwhelmed by the number of channels available to them. 83% of respondents watch 10 or fewer channels over and over again. That is down from 86% in 2013. Only 58% say that it’s easy to find something they ‘want’ to watch.

Pay-per-View Events: Only 10% of households have watched at least one PPV event, things like boxing or UFC fights (not movies), during the last year.

OTT Usage: 56% of households buy at least one OTT service like Netflix. 33% of households that buy OTT watch it more than 2 hours per day. 36% of households have used OTT per-rental services like Redbox or movies on Amazon Prime. 70% of those who use rental services watch content on a weekly basis. 80% of people using OTT report that it’s easy to find things they ‘want’ to watch.

TV Everywhere: Only 43% of respondents were aware that their cable provider offers TV Everywhere programming. Only 23% of respondents use TV Everywhere.

Social Media: 22% of respondents have posted on social media while watching TV. 34% have watched new programming based upon a recommendation from somebody they know on social media.

The Security / Privacy Battle

SpyVsSpyEvery time there is some traumatic terrorism event like what just happened in Paris there is a renewed call by governments for better surveillance and security measures. And every time that happens, the advocates of privacy sound a loud warning. What I find most interesting about this back and forth between the two sides is that it’s not events or even public policies that are driving the battle between security and privacy, but technology.

Just during the last decade there has been a number of technologies that have assaulted our privacy – encryption, big data, cloud computing, and advertising spyware. And we are fast approaching new threats from drones and from Internet of Things sensors everywhere.

The real battle between security and privacy happens when we introduce new innovations that can invade our privacy followed by countermeasures against those new technologies. There are plenty of politicians on both sides of the privacy issue who think that creating new laws is the way to protect privacy. But there are no laws that are going to flexible enough to keep up with the new threats we are constantly seeing in the real world.

Consider the traditional privacy laws. There have been wire-tapping laws on the books for decades which are now completely obsolete. The FBI convinced the FCC a few decades ago to create a set of laws called CALEA that gives the FBI the right to subpoena ISPs and get the records of suspected law breakers. ISPs and telcos spend a lot of money to stay compliant with these rules and yet I can’t think of one of my clients that has actually gotten a CALEA request from the FBI. ISPs do often get requests from local law enforcement asking for calling records under older wire-tapping laws, but not a peep out of the CALEA folks.

And this is because those laws were obsolete before the ink was dry on them. The CALEA rules were written not long after we had migrated from dial-up to DSL and there was no such thing as the dark web and disposable cell phones and all of the other ways that serious criminals use to avoid law enforcement.

What typically happens with a new technology is that it gives one side – the police or the bad guys – a temporary advantage. But there is always a technological counterpunch as somebody on the other side figures out how to defeat and neutralize each new technological development.

Edward Snowden showed us that law enforcement sometimes is so desperate for an edge that they collect data illegally in violation of the basic rights granted to US citizens by the fourth amendment. But even that is only a temporary edge. There are now numerous groups developing strategies to counteract widespread government surveillance.

There have been numerous attempts to pass surveillance and security laws starting with the Patriot Act. But industry experts say that most of the laws that try to give the government more power are ineffective, again because technology moves a lot faster than legislative bodies.

So what we see is a cat and mouse game. The NSA spies on us and so companies like Apple develop encryption that makes it hard or impossible for the NSA to gather anything useful. And there are more and more web services that either automatically encrypt or which offer that as an option.

It seems that the privacy advocates are winning the long term fight, and this is because there are ways around almost any tool the government or big business can use to spy on people. I’ve read several articles recently that talk about how even in China people are finding ways to bypass the strict security of the Great Firewall of China. But the fight is a long way from over because there are always going to be tools that come out that can be used to spy on people and there will then be ways to defeat those measures. We are likely to see this battle for decades to come.

Reform at the FCC

FCC_New_LogoThe House recently passed by voice vote H.R 3675, the Federal Communications Commission Process (FCC) Reform Act of 2015. That’s a little surprising in that it often feels like this Congress never passes anything in recent years.

This act looks to improve the processes at the FCC. It should be noted that many of the items included in this act came from recommendations arising out of an internal self-assessment taken by the FCC on their own processes.

Over the years, like often happens at many regulatory agencies, processes get tweaked until they become somewhat dysfunctional. Little changes made each year are usually minor, but over the years these changes accumulate. In the FCC this has resulted in administrative rules that have been used to stretch out timelines and to delay decisions.

The Act would require the FCC to seek comments from the industry and then adopt rules that would:

  • set minimum comment periods for rulemaking proceedings,
  • allow time for public comment by eliminating the practice of placing large amounts of data into the record on the last day of the public comment period,
  • increase transparency regarding items before the commissioners,
  • require publication of the text of proposed rules, and
  • set timelines for FCC action on certain types of proceedings.

The bill would also require that the FCC:

  • Publish the text of items before they are voted on.
  • Create a searchable database for consumer complaints.
  • Publish more documents on the FCC’s website.
  • Potentially allow more than two commissioners to meet outside of the formal meetings without violating the Sunshine laws.

I find the bill interesting because there has been a quiet war between Congress and the FCC for a decade. Some congressmen have tried to largely defund the FCC several times due to not liking the policy positions they have taken on things like net neutrality. And more globally, the current pro-big business congress generally dislikes federal regulatory agencies of any kind.

But to someone who follows the FCC the proposed changes look very reasonable. I’ve always found it troublesome that the FCC could start major inquiries or rulemakings and then could conceivably never make a decision on a topic. So requiring firm timelines is probably good for the industry and good for the country. The major complaint that carriers have about regulators is the uncertainty that hangs over them whenever major changes are being contemplated.

And as someone who writes an industry blog, opening up the FCC complaints to the public sounds like a treasure trove. I am imagining the big piles of stories sitting there about carriers’ bad behavior just waiting to be turned into juicy headlines. And as somebody who occasionally makes an FCC complaint there now might be the chance that somebody might read one of them. I do have to wonder how they are going to stop parties from filing comments on the last day of a public comment period – the tool of procrastinators everywhere.

There are already some processes at the FCC that work pretty well. I love that the FCC must publish the minutes and show the public any documents that were given to them in ex parte meetings. But it would be even better if more of the process of deliberating big topics was made public. The big carriers all have swarms of lobbyists who know about everything going on at the FCC. But my smaller clients are often caught by surprise by FCC rulings and they’ve often wished that they would have had more of a chance to provide input to the deliberation process.

Overall this looks like a positive change, assuming that Congress actually gives the FCC enough money to do what they are being told to do.

 

AT&T’s Vision of the Future in 1993

Bell_logo_1969This link is to some old AT&T commercials from 1993. You might remember these as visions of future technology that AT&T was going to bring us. It’s really interesting to look at their predictions and see how they did. I assume that in 1993 that AT&T probably had folks in Bell Labs looking at all of these ideas.

To put these predictions into perspective, 1993 is thought of now as the launch date of the public web. That’s the year that the Mosaic browser was launched that allowed for the combination of text and graphics and the creation of web pages. Before then the web was still something used mostly by techies and there was very little appreciation for how quickly this would explode onto the market. Here are a few of the more interesting claims from AT&T in 1993:

Borrowed a book from thousands of miles away. The video shows somebody viewing a scanned book. AT&T missed the popularity of e-readers but generally got this right.

Driven across the country without stopping for directions. The US government didn’t release commercial access to the GPS system until 2000. AT&T obviously believed this was going to happen.

Sent someone a fax from the beach. I don’t think AT&T can be faulted too badly for missing the popularity of email and PDF files.

Paid a toll without slowing down. This was not a daring prediction since Norway had implemented the first electronic toll collection system in 1991.

Bought concert tickets from a cash machine. It’s funny to think of having to stand in line at an ATM to buy things on the web. I’m already always annoyed at a person who doesn’t know how to use an ATM and I can’t imagine standing behind somebody who is shopping.

Tucked your baby in from a phone booth. AT&T tried numerous times to push picture phones and it never caught on with the public. They envisioned video phone booths talking to AT&T video phones in the home. In 1993 it was probably hard to envision Skype and FaceTime and the near death of phone booths.

Opened doors with the sound of your voice. This can be done now but is not very popular. But opening your door with a smartphone is starting to gain a little traction.

Carried your medical history in your wallet. This never happened and for various privacy reasons hasn’t even been done very well yet in the cloud from the doctor’s office. You still have to fill out forms for half an hour to see a new doctor.

Attended a meeting in your bare feet. This was a prediction of Skype, but using a picture phone instead of a PC or smartphone.

Watched a movie you wanted the minute you wanted. They foresaw digital transmission of video. 1993 saw the very early trials of cable modems, but DOCSIS cable modems and DSL did not become commercially viable until the end of the 1990s. But AT&T obviously believed that data speeds would get much faster.

Learned special things from far-away places. This supposed interactive distance learning which has become common with the web and which many think will eventually be the primary form of education.

Overall AT&T didn’t do too badly in their predictions, and as the company that owned Bell Labs you would expect them to have insight to the next decade of technology. But they did fail to forecast that cable companies would kick their butts in the marketplace and become the dominant data providers.

Broadband Shorts for December

fast fiberFollowing are a few topics I found interesting but which are too short for a whole blog.

Petabit Fiber Speeds: Bell Labs announced a successful trial in the lab of 1-petabit speeds on a fiber (that is 1,000 terabits). This was done by using real-time space-division multiplexed optical multiple-input-multiple-output (MIMO-SDM) technology. The technology is able to send six separate signals through the fiber without having them interfere with each other. Within each of those signals is another full set of other signals that utilize separate light frequencies.

It always take a few years to go from lab to finished product but this is a major breakthrough with the fastest commercial fiber systems operating at 20 terabits. This is a 50 times increase in capacity. This kind of technology would be used on long-haul routes to move massive amounts of data between two points. The lasers and electronics for this are bound to be very expensive, but in places where that much capacity will be needed it will probably be much cheaper than operating large numbers of fiber pairs. In a world where data usage is growing at a geometric rate, a 50-times increase in capacity only moves us a decade or two ahead of demand.

Comcast Data Caps: A leak of internal documents used to train customer service reps show that Comcast has dropped all pretense that their new ‘trial’ of data caps is based upon network congestion. In fact, Comcast is now training their employees to emphatically deny that congestion is an issue and instead wants customer service reps to tell customers that caps are all about ‘fairness’ and of offering ‘a more flexible policy’ for customers.

I could buy that new story line if they were using the caps to somehow give customers the ability to buy a cheaper connection by agreeing to have a cap. That would indeed be flexible and fair to the small users. But it’s hard to see any flexibility where nobody’s price goes down but customers that actually use the data to which they have subscribed must pay more. But like most huge companies. Comcast is now in full double-speak mode and telling customers the exact opposite of what they are actually doing.

Fewer Ads on Cable: While cable companies will not publicly acknowledge that their major competition might be Netflix, a number of major content providers like Time Warner, Fox, and Viacom are quietly cutting back on the number of ad slots they cram into a given hour of television.

For example, the CEO of Time Warner pledged to try to cut advertising slots during prime time in half – a major reduction. They will start with a trial after New Year’s with TruTV which carries programming that is largely reality TV aimed at younger viewers. If the trial is successful they plan to move this same idea to other networks like TBS, TNT, and CNN.

It’s becoming obvious that the average person is tiring of intrusive advertising. The spread of ad blockers on the web shows how much people hate ads. As a cord cutter I have almost entirely eliminated video ads from my life and it feels like this has given me more time (which of course I then use to watch more time-wasting programming – but it still feels good).

Media Usage by Kids: Common Sense Media did a major survey of the media usage of kids and gave us a detailed look into how kids use various kinds of digital media. For instance, they found that tweens (kids between 8 and 12) use digital media of some sort an average of six hours per day. This might be streaming music, watching television and videos, using social media, playing games, texting, or posting to web sites. Older teens use digital media over 9 hours per day.

Kids are often using digital media in the background while doing other things, so they might not be concentrating on it the whole time, but have it on in the background. Tweens still use television as their most common digital activity. But for older teens music has bypassed television.

25% of teens who go online say that their parents don’t understand what they do online. 30% say that their parents don’t understand or know about the social media they use. 53% of teens and 72% of tweens say that their parents have talked to them about the time they spend using media and the content they view.

The biggest problem identified in the study was the continuing digital divide. As schoolwork goes online, kids without adequate broadband are finding it impossible to keep up with kids that have access. The report showed that 10% of lower-income kids still have dial-up Internet access and only half of lower-income kids have smartphones – both very different statistics than kids from more affluent households.

OTT By the Numbers

roku-3-2Lately I’ve been reading about how much OTT video services have grown and so I looked to see how big the phenomenon has become. What I found was that you get a different answer depending upon who is doing the counting and how you define OTT.

It’s easy to start with Netflix, which is clearly the largest provider of alternate programming. As a public company they publish their subscriber numbers every quarter and at June 30 of this year they had 29.8 million paid subscriptions in the US and had added 605,000 customers in the quarter. With approximately 134 million housing units in the country that’s a penetration rate of over 22%.

Parks Associates tracks the OTT industry and they recently released a list of the top 10 OTT companies, ranked by the number of paid subscribers. They say there are over 100 pay OTT services available in the US. The top 10 list is interesting and probably includes things that many people aren’t aware of or that don’t think of as OTT. The most recent top 10 list is as follows:

  1. Netflix
  2. Amazon Video
  3. Hulu
  4. MLB TV
  5. WWE Network
  6. HBO Now
  7. Crunchyroll
  8. NFL Game Pass
  9. The Blaze
  10. Sling TV

I think most people would have guessed the top 3. It’s interesting that 3 of the top 10 are sports networks. The subscriber numbers for baseball and football are very seasonal and move up and down the list depending upon the time of the year. Major League Baseball (MLB TV) just announced that starting next season they will only make their programming available to subscribers of a cable service, so they will fall off this list. Crunchyroll features Japanese anime, manga, and auto racing. The Blaze includes Glenn Beck and various other political shows. To show how low the threshold for getting on the list is, number 10 is now Sling TV that just started early this year and currently claims about 400,000 subscribers.

Using the above definition of OTT, Parks Associates reports that 58% of US broadband households have used at least one OTT video service in the past 30 days. They say that a little more than 25% have used two or more different OTT services.

But there are others counting OTT using a wider definition. For instance, the numbers jump way up if you include things like YouTube, which has more viewers than Netflix. The multiservice screen provider Clearleap reports that when counting services like YouTube 71% of households report using OTT services. This count differs from the Parks Associates count by also considering smartphone-only usage rather than only considering homes with a broadband connection.

There are a number of other surveys around also and each differs in defining what is included as OTT and also differ by the type of platform used to watch the content. So any time you see OTT statistics it’s important to dig a bit to understand just what and who is being counted.

One thing that all of the surveys agree on is that younger people view a lot more video than anybody else. Common Sense Media just reported that teens between 13 and 18 use an average of 9 hours per day of entertainment media. This would include not only OTT content, but normal TV, on-line games, social media, and sites like Vine which are not counted as OTT but which include video content. As a parent of a teen I would say that number sounds just about right.

The Continued Growth of Broadband

Broadband sales continue to boom at Comcast and Time Warner and the largest cable companies as a group added over 787,000 customers in the third quarter of the year. The largest telcos lost 143,000 customers for a net increase of 644,000 new broadband customers for the quarter. Not included in these numbers are the small telcos and cable companies as well as the FTTP providers and municipalities.

The counts for the largest companies are as follows for the quarter:

‘                                                           End of 3Q                   New in 3Q

Comcast                                              22.9 M                         320,000

Time Warner Cable                            13.0 M                         246,000

Charter                                                  5.4 M                         147,000

Cablevision                                           2.8 M                              3,000

Suddenlink                                            1.2 M                           21,600

Mediacom                                             1.1 M                           16,000

WOW                                                      0.7 M                              (800)

Cable ONE                                             0.5 M                              (171)

Other large cable companies              6.7 M                          35,000

Total Large Cable                                54.3 M                        787,629

AT&T                                                     15.8 M                      (129,000)

Verizon                                                   9.2 M                            2,000

CenturyLink                                           6.1 M                         (37,000)

Frontier                                                  2.4 M                          27,000

Windstream                                           1.1 M                        (11,200)

FairPoint                                                 0.3 M                          (1,338)

Cincinnati Bell                                       0.3 M                            6,200

Total Large Telcos                                35.2 M                      (143,338)

Total Large Companies                       89.5 M                        644,291

You have to be a little careful in looking at the telco numbers because many of those companies are seeing increases in higher bandwidth products while seeing continuing erosion in the older and slower DSL products. For example, Verizon FiOS added 114,000 fiber customers for the quarter but lost almost the same number of DSL customers. AT&T said they gained 172,000 IP broadband customers, which is a combination of those on U-verse and fiber.

The 89.5 million broadband customers of these large companies represent over 66% of all households in the country (135 million total households). These companies actually serve more customers than shown. For example, there are many cases where these companies sell broadband to an apartment owner that then distributes it to the apartment units.

For the cable companies the quarterly increase of 787,629 is an annual growth rate of 5.8% and will annualize out to over 3 million new broadband subscribers. It’s obvious that more and more households are finding it mandatory to have a broadband connection and the whole market keeps creeping upward.

Why No Redundancy?

Copper wireI usually load a blog every morning between 7:00 and 8:00 eastern. But today my Internet was down. I first noticed then when I woke up around 2:30. Don’t even ask why I was up then, but that is not unusual for me. My Internet outage was also not that unusual. I have Comcast as my ISP and they seem to go out a few times per month. I’ve always given them the benefit of the doubt and assumed that a few of the late night outages are due to routine network maintenance.

So I grab my cell phone to turn on my mobile hot spot. Most of the outages here last an hour or two and that is the easy way to get through outages. But bam! – AT&T is out too. I have no bars on my LTE network. So my first thought is cable cut. The only realistic way that both carriers go out in this area is if the whole area is isolated by a downed fiber.

I check back and hit a few web sites and I find at about 3:00 that I have a very slow Facebook connection, but that it’s working. I can get Facebook updates and I can post to Facebook, but none of the links outside of Facebook work. And nothing else seems to be working. This tells me that Facebook has a peering arrangement of some kind with Comcast and must come into the area by a different fiber than the one that was cut.

So I start looking around. The first thing I find is that Netflix is working normally, just as fast as ever. So now I have a slow Facebook feed and fast Netflix and still nothing else. After a while Google starts working. It wasn’t working earlier, but it seems that I can search Google, although none of the links work. This tells me that Comcast peers with Google but that the Google links use the open Internet. I force a few links back through the Google URL just to see if that will work and I find that I can read links through Google. No other search engines seem to be working.

The only other think I found that worked with the NFL highlight films and I was able to see the walk-off blocked punt in last night’s Ravens – Browns game. It’s highly unlikely that the NFL has a peering relationship with anybody and they must have a deal with Google.

So now I know a bit about the Comcast Network. They peer with Netflix, Google and Facebook – and since these are three of the largest traffic producers on the web that is not unusual. And at least in my area the peering comes into the area on a different fiber path than the normal Internet backbone that has knocked out both Comcast and AT&T.

But I also now know that in my area that Comcast has no redundancy in the network. I find this interesting because most of my small clients insist on having redundancy in their networks. Of course, most of them operate in rural areas that are used to getting isolated when cables get cuts – it happened for many years with telephone lines and now with the Internet.

But I can see that Comcast hasn’t bothered creating a redundant network. This particular outage went for 7 or 8 hours which is a bit long, so this must be from a major fiber cut. But I look at a map of Florida and it is a natural candidate to have rings. Everybody lives on one of the two coasts and there are several major east-west connector roads. This makes for natural rings. And if our backbone was on a ring we wouldn’t even know there was an outage. But with all of their billions of dollars of profits, neither Comcast nor AT&T wireless cares enough about redundancy to have put our area backbone on a ring.

And I also don’t understand why they don’t have automatic alternate routing to bypass a fiber cut. If Netflix, Facebook and Google were connected everything else could have been routed along those same other fibers. That is something else my clients would have done to minimize outages for customers.

This is honestly unconscionable and perhaps it’s time we start clamoring to the FCC to require the big companies to plow some of their profits back into a better network. These same sort of outages happened a few times to the power grid a decade ago and the federal response was that the electric companies had to come up with a better network that could stop rolling outages. I know some of my clients that are electric companies spent some significant dollars towards that effort, and it seems to have worked. Considering how important the Internet has become for our daily lives and for commerce perhaps it’s time for the FCC to do the same thing.