Current News Technology

The FCC and Peering

Zeus_peering_around_a_corner__(9386751334)As the politics of net neutrality keep heating up, Senator Pat Leahy and Representative Doris Matsui introduced the Online Competition and Consumer Choice Act of 2014.

This bill requires the FCC to forbid paid prioritization of data. But then, Senator Leahy was quoted in several media outlets talking about how the bill would stop things like the recent peering deal between Netflix and Comcast. I’ve read the proposed bill and it doesn’t seem to ban those kinds of peering arrangements. His comments point out that there is still a lot of confusion between paid prioritization (Internet fast lanes) and peering (interconnection between large carriers). The bill basically prohibits ISPs from creating internet fast lanes or in disadvantaging customers through commercial arrangements in the last mile

The recent deals between Netflix and Comcast, and Netflix and Verizon are examples of peering arrangements, and up to now the FCC has not found any fault with these kinds of arrangements. The FCC is currently reviewing a number of industry peering agreements as part of investigating the issue. These particular peering arrangements might look suspicious due to their timing during this net neutrality debate, but similar peering arrangements have been around since the advent of the Internet.

Peering first started as connection agreements between tier 1 providers. These are the companies that own most of the long haul fiber networks that comprise the Internet backbone. In this country that includes companies today like Level3, Cogent, Verizon and AT&T. And around the world it includes companies that you may not have heard of like TeliaSonera and Tata. The tier 1 providers carry the bulk of the Internet traffic and peering was necessary to create the Internet as these large carriers need to be connected to each other.

Most of the peering arrangements between the tier1 carriers have been transit-free or what is often referred to as bill-and-keep. The traffic between the major carriers tends to balance out in terms of originating and terminating volumes and in such cases it doesn’t make a lot of sense for two carriers to bill each other for swapping similar amounts of data traffic.

But over time there were peering arrangements made between the tier 1 carriers and tier 2 providers that includes the large ISPs and telcos. Peering was generally done in these cases to make the network more efficient. It makes more sense to interchange traffic between and ISP and somebody like Level3 at a few places rather than at hundreds of places. It’s always been typical for these kinds of peering arrangements to include a fee for the tier 2 carrier, something that is often referred to as a transit fee.

There is no industry standard arrangement for interconnection between tier 1 and tier 2 providers. And this is because tier 2 providers come in every configuration imaginable. Some of them own significant fiber assets of their own. Others, like Netflix have a mountain of one-directional content and own almost zero network. And so tier 2 providers scramble to find the best commercial arrangement they can in the marketplace. One thing that is almost universal is that tier 2 providers pay something to connect to the Internet. There is no standard level of payment and transit is a very fluid market. But payment generally recognizes the relative level of mutual benefit. If the traffic between two parties is balanced then the payments might be small or even free. If one party causes a lot of costs for the other then payments typically reflect that imbalance.

Netflix has complained about paying Comcast and Verizon. But those ISPs wanted payments from Netflix since the traffic from Netflix is large and totally one-directional. Comcast or Verizon needs to construct a lot of facilities in order to accept the Netflix traffic and they don’t get any offsetting benefit of being able to send traffic back to Netflix on the same connection.

In economic terms, on a national scale the peering market is referred to as an n-dimensional market, meaning that a large tier 2 provider has the ability to negotiate with multiple parties to achieve the same result. For example, Verizon has a lot of options for moving data from the east to the west coast. But eventually the Internet becomes local, and that is where the cost and the contention arises. As Internet traffic enters a local metropolitan market it begins to hit choke points where the traffic can overwhelm the local facilities and cause congestion. The payments that Comcast or Verizon want from Netflix are to build the facilities needed for getting Netflix movie traffic to and through these local hubs and chokepoints.

Peering arrangements like this make sense. I find it hard to believe that the FCC is going to get too deeply involved in peering arrangements. It’s an incredibly dynamic market and carriers are constantly rearranging the network as they find better prices or more efficient network arrangements. If there is any one place where the market works it is between the handful of large carriers that handle the majority of the Internet traffic. Most of the bad things that can happen to customers are going to happen in the last mile network, and that is where net neutrality should properly be focused.

And why the picture of the kitten? I work at home and at my very local part of the network this is the kind of peering that I often get.

Current News

Politics and Net Neutrality

Politics is always around in our industry but it is mostly out of sight. The big telecom companies maintain hordes of lobbyists to push their interests, but this is mostly done out of the public eye. It’s been a rare thing during my career to see a telecom issue play out big in the news and it’s only happened a few times. I remember a flurry of politics during the passing of the Telecommunications Act of 1996. But I’m not sure many people outside the industry paid a lot of attention to that. I remember some louder politics that same year when they passed the Communications Decency Act that tried to get pornography off the Internet. But for most of my career it has been rare to have politics intersect visibly with the industry.

Our industry makes the news fairly often, but the headlines are usually about things like mergers rather than about politicians debating both sides of a telecom issue. But net neutrality has grown to be an issue that is in the news every week. Net neutrality was political news once before, but it’s very different this time around. In 2011 republicans tried to pass a bill to repeal the FCC’s net neutrality rules. But that bill never got a lot of traction and nobody outside of the industry probably even heard about it.

All of a sudden net neutrality is being discussed everywhere. It’s even made it into popular culture. Stephen Colbert and John Stewart both launched into funny diatribes in favor of net neutrality. John Oliver went on a 13-minute rant about net neutrality and got so many people to contact the FCC that it crashed their servers. The advocacy group The Free Press along with 85 other organizations delivered over a million signatures on a petition to the FCC asking then to enforce net neutrality. That’s a lot of signatures and Google shows that only a few other petitions have gotten that many signatures, including one earlier this year in protest of the Russian figure skating judges in the Olympics. This is getting into rarified air in the world of popular culture.

Polls seem to make it clear that the majority of people don’t want the big carriers to mess with the Internet. Yet, perhaps sadly, politicians are weighing in on net neutrality straight down party lines like with so many other issues these days. Earlier this year a number of house republicans sent a petition to the FCC asking them to halt any considerations of imposing net neutrality rules. Last week the democrats in the house and senate proposed bills that would prohibit the FCC from allowing ‘fast lanes’. You can’t look at political news lately without seeing another politician saying something about net neutrality.

I don’t know what to make of all of this. As somebody who works in the industry I generally hope that we are able to work out our own issues in the normal fashion – which is to have the FCC issue a new order and then have the courts decide which parts of the provisions are legal and sustainable. It’s a bit of an awkward system because it often takes a few years between first order and final implementation, but it mostly has been working.

Right now because of the court order overturning the FCC’s original net neutrality order we are operating in a vacuum on the issue. There are no rules in place at the FCC that require or ban most carrier practices in this area. We instead have some vague rumblings from FCC commissioners telling carriers to not do anything too outrageous or they will face some unknown consequences.

Perhaps I should take some solace that we currently have a split House and Senate, each controlled by a different party. This puts net neutrality in political limbo and it is highly unlikely that either party will be able to do anything about the topic from a legislative perspective.

But I am not comforted by that limbo, because my fear is that over the course of a decade or so that each of the parties might have a time where they have enough votes to change the net neutrality rules to their liking. I envision one set of rules being put in place by one party and then those rules overturned when the next party gets into power. What this means in practical terms is that the industry will be in limbo over the topic for a long time, never quite able to trust whatever rules are in place at any given time. And the one thing I have seen in this industry is that uncertainty is a bad thing. Uncertainty in this industry often ends up getting manifested by cutbacks in capital spending in the areas of concern. The last thing we need is for carriers to be worried about making the investments needed to keep the Internet fast. Because if that happens, we all lose and net neutrality won’t be that important if the whole Internet gets impaired.

Current News Technology

Is There a Web Video Crisis – Part IV and Final

In the previous three installments of this blog I looked at the issues behind the demands of Comcast and Verizon to charge content providers for creating an Internet ‘fast lane’. In particular I have focused on the recent actions between Comcast and NetFlix. In everything I have read about this issue I never saw any specific reason cited why Comcast thought they needed the extra payments from NetFlix, and this blog series has been about looking for such reasons.

In the earlier blogs I looked at the various components of the Comcast network and my conclusion is that end-user customer fees ought to be covering the cost of the wires, or at least that is how all of the companies smaller than Comcast and Verizon see the issue. I then looked at the issue of preparing the network for peak video usage during simulcasts. Again, my conclusion is that this is a function that is a normal part of making your network operational and doesn’t seem like a reason to charge a premium price to get what is supposed to be there. Finally, I looked at peering, data centers and the network of routers and switches. My conclusion there was that peering generally saves money for Comcast and Verizon and that their savings from peering are far larger than their costs.

In the months leading up to the announcement that the two parties had reached a deal, I had seen numerous complaints from customers who said that their NetFlix was not working well on both Comcast and Verizon. And there were numerous articles like this one asking if Comcast and Verizon were throttling NetFlix. There was clearly something fishy going on and it and it was clear that both Verizon and Comcast were somehow slowing down NetFlix bits as compared to other bits. The complaints were all coming from NetFlix traffic and we didn’t see the same complaint about AmazonPrime or other video providers. And I heard no complaints anywhere about the speeds on the TV Anywhere products offered directly by Comcast and Verizon. I know I was watching Game of Thrones online in HD through my Comcast subscription and it always worked perfectly.

Then, when there was an announcement, it was made to sound like NetFlix was the one who was requesting premium access from Comcast. The Verizon deal was done much more quietly and there was no similar insinuation there. But almost instantly after Comcast struck the deal with NetFlix the speeds popped back up to former levels

One has to ask if NetFlix really got premium treatment of their bits or if Comcast simply removed whatever impediments were slowing them down. I will be the first to admit that I, like almost everybody else, am an outsider and we really don’t know what the two parties discussed as part of this announcement. But when I look at the facts that are known to me, what I see is that Comcast and Verizon were flexing their monopoly powers and slowing NetFlix down to extract payment out of them

There is no doubt that the NetFlix traffic causes cost to these two companies. Video traffic has been growing rapidly on the Internet and NetFlix is the largest single provider of video. But I step back and have to ask the basic question of what end-user fees for Internet are supposed to cover. A customer pays for a connection of a given speed, and it seems to me like these companies have promised a customer that they could use that speed. There is the caveat that Comcast has a data cap – a topic of another blog – but as long as a customer stays under that data cap they ought to always get the speed they have purchased. It shouldn’t matter if that customer chooses to use that speed and capacity to watch NetFlix or read silly telecom blogs – they have paid for a certain level of performance.

For Comcast to say that their network is not capable of delivering the accumulated speeds they have sold to customers sounds to me like they have oversold the capacity of their network. They want customers to buy fast speeds, but they don’t actually want them to use it. I’m not a lawyer, but this starts sounding like fraud, or something similar to fraud.

I simply don’t understand why the FCC would listen to any argument that says that content providers have to somehow pay extra to get normal performance. Because that is what it looks like NetFlix had to do. I can imagine as part of that agreement that there was a nondisclosure signed of the terms, and this NetFlix is not out yelling like they probably ought to be

But the long-term results of what Comcast and Verizon have done is that end users are going to pay twice for video access. They already pay to get a data pipe which is large enough to receive video. And now the cost of movies or movie subscriptions is going to increase to cover what NetFlix has to pay to deliver those movies. NetFlix is certainly not going to eat such costs.

And so the consumer is being screwed by a clear case of corporate greed. I have come to the conclusion that Comcast extracted payments out of NetFlix simply because they are large enough to do so. That is an abuse of monopoly power, and that power is only going to get worse if they are allowed to buy Time Warner.

Current News Regulation - What is it Good For?

Net Neutrality Enters the Twilight Zone

In the telecom world we are not very used to our issues getting a lot of notice from the public. But it’s obvious that net neutrality has become a political issue as much as it is an industry issue. Compared to the normal way we do business as an industry the debate has entered the twilight zone. This all got started when new FCC Chairman Tom Wheeler said that he was proposing new rules that would allow for the creation of an Internet ‘fast lane’, By that he meant that the FCC is going to allow the large ISPs to charge large content providers for premium access to their networks.

Of course, Chairman Wheeler is not himself neutral in this decision having spent years as the head lobbyist for the cable industry and opposing net neutrality. It’s somewhat ironic that he made this new announcement at the annual cable show with his cable company peers. The headlines that day made it sound like the FCC was going to take a legitimate shot at maintaining net neutrality, but within days it became understood that the fast lane idea was just the opposite and that he was handing the cable companies exactly what they wanted.

What I don’t think that Wheeler expected was that the public would jump all over his idea. And so, before the proposal was even released the Internet companies like Google and NetFlix weighed in against it. A huge number of consumer groups and many citizens weighed in against it.

And so, quite unexpectedly, the Chairman announced yesterday that he is changing the proposed rule, one that hasn’t even been released yet. He said that the revised rules would allow for ISPs to charge companies like NetFlix and Amazon for faster access to customers, but that non-paying companies would not be put into the slow lane. This makes no sense and is political double-speak. From a network engineering perspective you either give priority to bits or you don’t. If some companies get priority routing, then all other traffic gets degraded. That is the only way it can work on a network and no amount of regulatory talks can change the way that bits operate.

The idea gets even more bizarre if you think it through. What happens if 20 companies pay Comcast for priority access? Does the one who pays the most get slightly more priority than number two, and so on? The fact is that networks can’t do that. Bits are either prioritized or they are not, and so if a lot of companies pay for priority access we end up back where we are today for those companies, while the rest of the Internet would get degraded service.

One thing that pushes this into the Twilight Zone is that Rasmussen did a push poll on the topic and concluded that only 21% of Americans are in favor of net neutrality. Push polls are generally only used for hot button political topics where somebody wants to prove the opposite of what’s true. In this case, the main question of the poll was, “Should the FCC regulate the Internet like it does radio and television”. None of the questions asked had anything to do with net neutrality and instead were designed to elicit a specific negative response. Obviously there are dozens of better ways to have asked the public about net neutrality, including actually asking about it.

I have not conducted a poll, but I traveled all last week and in conversation I asked a number of people what they thought about the idea that the ISPs could give some companies priority access, which implies that others would get something less. Nobody thought that was a good idea and the general consensus was to leave things working the way they are. I believe there will be a huge amount of public discontent should the ISPs be allowed to break the Internet.

I don’t think Chairman Wheeler has any comprehension how important the Internet is to most people. He is skirting with making a huge blunder if he allows the Internet to get screwed up. He is making himself the public face of how the Internet functions, and if he breaks it people will blame him personally. He has the chance to become the next infamous political appointee to get compared to Michael Brown who was running FEMA during Hurricane Katrina. But perhaps he won’t mind being vilified since he is handing the cable companies a billion dollar opportunity to charge more to Internet companies.

Current News

A Solution for Net Neutrality?

Today Mozilla filed comments with the FCC with a clever solution that would fix the net neutrality fiasco. Attached is the Mozilla filing. I call the solution clever, because if the FCC wants to solve net neutrality Mozilla has shown them a path to do so.

Mozilla has asked to split Internet traffic into two parts. First is the traffic between ISPs and end-user customers. Mozilla is suggesting that this part of the business can remain under the current regulatory rules. The second portion is the traffic between ISPs like Comcast and AT&T and content providers like Facebook, NetFlix, etc. Mozilla recommends that the FCC reclassify this as transport under Title II of the Telecommunications Act of 1996.

The current dilemma we are facing with net neutrality is that FCC lacked the courage to classify the Internet network as common carrier business. Instead, in 2002, when broadband was growing explosively, the FCC classified all Internet traffic as an information service. And that decision is why we are even having the debate today about net neutrality. If the FCC had originally decided to regulate the Internet then it would have full authority to enforce the net neutrality rules it passed a few years ago.

But even in 2002 the FCC was a bit cowed by the political pressure put on them by lobbyists. The argument at the time was that the FCC needed to keep hands off the burgeoning Internet so as to not restrict its growth. It’s hard for me to see how classifying the Internet business as common carrier business would have changed the growth of the Internet and I believe it all boiled down to the fact that the cable companies did not want to be further regulated by the FCC.

The net neutrality rules written a few years ago by the FCC basically say that ISPs have an obligation to deliver all packets on the Internet without discrimination. Mozilla is suggesting that there is an additional legal obligation between ISPs and content providers to deliver their traffic without discrimination.

This argument might seem a bit obscure to somebody not in the industry, but it removes the dilemma of not being able to regulate the traffic between ISPs and content providers. The suggested change is to not classify data packets at the carrier level as information services, but to recognize it by its normal network function – that is the transporting of data from one place to another. Today transport is regulated in the sense that if a carrier sells a data pipe of a certain amount of bandwidth to another carrier they are obligated to deliver the bandwidth they have charged for. By putting the gigantic data pipes that extend between companies like NetFlix and Comcast under the transport regime it would treat Internet traffic like any other data pipe.

This change makes a lot of sense from a network perspective. After all, it’s hard to think of the transaction where NetFlix hands a huge data pipe to Comcast or AT&T as an information service. Comcast is doing no more than taking the data on that pipe and moving that data where it is supposed to go. That is the pure definition of transport. It only becomes an information service on the last mile of the network where the data traffic is handed off to end-user customers. There are already millions of other data circuits today that are regulated under the transport rules. It make logical sense to say that a 10 gigabit Internet circuit is basically the same, at the carrier level, as a 10 gigabit circuit carrying voice or corporate data. Data pipes are data pipes. We don’t peer into other data pipes to see what kind of traffic they are carrying. But by classifying the Internet as an information services that is exactly what we do with those circuits.

This idea gives the FCC an out if they really want net neutrality to work. I personally think that Chairman Wheeler is thrilled to death to see net neutrality being picked apart since he spent years lobbying against it before taking the job. So I am going to guess that the Mozilla suggestion will be ignored and ISPs will be allowed to discriminate among carriers, for pay. I hope he proves me wrong, but if he ignores this suggestion then we know he was only paying lip service to net neutrality.

Current News

Looking Into 2014

Crystal ball Français : Boule de cristal (Photo credit: Wikipedia)

As any year comes to a close it’s always fun to look forward to the next year and to make some guesses about the direction of our industry. I have always done this, but this will be the first year I put my guesses out in public with a blog. I plan to come back at the end of next year to see how I did.

More Consolidation of the Big Players. While the new FCC Chairman says that he is pro-competition, I think 2014 is going to see a lot of consolidation among the big players in the industry, which will lead to less competition. It’s likely that there will be major acquisitions in both the cellular and the cable TV space that will reduce the number of major companies in each industry. One has to wonder at what point the FCC will say no to acquisitions, but I don’t think 2014 is going to be that year.

AT&T’s Response in Austin Will Squelch Major Market Competition. I predict that AT&T’s announcement that they will build fiber-to-the-premise to match what Google is doing there is going to kill competition in NFL cities for a while. I don’t expect any new major announcements of plans to build NFL cities in the coming year. There will still be new FTTP overbuilders in smaller markets, but everybody is going to be gun-shy against committing money to major markets.

Network Neutrality Will Erode. The FCC is going to follow the lead of the new Chairman and will support large company initiatives to weaken network neutrality. This might be done through inaction, in that some large carriers may make arrangements to give preferential treatment in the network and the FCC may fail to halt the practice.

Transition to All-IP Network Will Creep Forward. While the large telcos all would like the transition to happen overnight, it’s probably going to take 3 – 4 years for a transition of most of the POTs network to IP. However, there will be some major steps taken in 2014 to start defining the regulatory framework that will go along with an all-IP network.

The Large Telcos Will Continue to Shed Copper Networks. The large telcos have made it clear that they would like to get out of the copper business. AT&T’s recent decision to bail on Connecticut is just the beginning. I think all that is probably stopping telcos from shedding more copper immediately is the lack of companies capable of buying large numbers of customers. But there will be more piles of customers shed in the next year.

The Smart Phone Will Begin to be the Hub For the Internet of Things. The main thing lacking for the Internet of Things to leap forward is consolidated platform to bring devices together. While there is the chance that some sort of home platform could eventually win this battle, I think 2014 is the year when more and more IoT devices are integrated with smartphones as the hub. If smartphones capture this role early they will be the de facto hub for a decade to come.

Customers Will Bail on Cable Faster Than Predicted. The phenomenon of households dropping or downsizing cable subscriptions will pick up steam this year and will go faster than predicted by the cable companies. The industry is not going to implode, but it will become obvious by the end of the year that there has to be a new paradigm created for delivery of programming and that traditional cable bundles cannot be the only product offered. It is going to take five years for the current cable model to break, but 2014 will be the year when the erosion becomes obvious.