Tag Archives: Internet traffic

How Vulnerable is the Internet?

OLPC: XO internet access

OLPC: XO internet access (Photo credit: Wikipedia)

A question you hear from time to time is how vulnerable the US Internet backbone is in terms of losing access if something happens to the major hubs. The architecture of the Internet has grown in response to the way that carriers have decided to connect to each other and there has never been any master plan for the best way to design the backbone infrastructure.

The Internet in this country is basically a series of hubs with spokes. There are a handful of large cities with major regional Internet hubs like Los Angeles, New York, Chicago, Dallas, Atlanta, and Northern Virginia. And then there are a few dozen smaller regional hubs, still in fairly large cities like Minneapolis, Seattle, San Francisco, etc.

Back in 2002 some scientists at Ohio State studied the structure of the Internet at the time and said that crippling the major hubs would have essentially crippled the Internet. At that time almost all Internet traffic in the country routed through the major hubs, and crippling a few of them would have wiped out a lot of the Internet.

Later in 2007 scientists at MIT looked at the web again and they estimated that taking out the major hubs would wipe out about 70% of the US Internet traffic, but that peering would allow about 33% of the traffic to keep working. And at that time peering was somewhat new.

Since then there is a lot more peering, but one has to ask if the Internet is any safer from catastrophic outage as it was in 2007? One thing to consider is that a lot of the peering happens today at the major Internet hubs. In those locations the various carriers hand traffic between each other rather than paying fees to send the traffic through an ‘Internet Port’, which is nothing more than a point where some carrier will determine the best routing of the traffic for you.

And so peering at the major Internet hubs is great way to save money, but it doesn’t really change the way the Internet traffic is routed. My clients are smaller ISPs, and I can tell you how they decide to route Internet traffic. The smaller ones find a carrier who will transport it to one of the major Internet hubs. The larger ones can afford diversity, and so they find carriers who can carry the traffic to two different major Internet hubs. But by and large every bit of traffic from my clients goes to and through the handful of major Internet hubs.

And this makes economic sense. The original hubs grew in importance because that is where the major carriers at the time, companies like MCI and Qwest already had switching hubs. And because transport is expensive, every regional ISP sent their growing internet traffic to the big hubs because that was the cheapest solution.

If anything, there might be more traffic routed through the major hubs today than there was in 2007. Every large fiber backbone and transport provider has arranged their transport networks to get traffic to these locations.

In each region of the country my clients are completely reliant on the Internet hubs. If a hub like the one in Dallas or Atlanta went down for some reason, ISPs that send traffic to that location would be completely isolated and cut off from the world.

There was a recent report in the Washington Post that said that the NSA had agents working at only a handful of major US Internet pops because that gave them access to most of the Internet traffic in the US. That seems to reinforce the idea that the major Internet hubs in the country have grown in importance.

In theory the Internet is a disaggregated, decentralized system and if traffic can’t go the normal way, then it finds another path to take. But this idea only works assuming that ISPs can get traffic to the Internet in the first place. A disaster that takes out one of the major Internet hubs would isolate a lot of towns from the region around it from having any Internet access. Terrorist attacks that take out more than one hub would wipe out a lot more places.

Unfortunately there is no grand architect behind the Internet that is looking at these issues because no one company has any claim to deciding how the Internet workd. Instead the carriers involved have all migrated to the handful of locations where it is most economical to interconnect with each other. I sure hope, at least, that somebody has figured out how to make those hub locations as safe as possible.

Should You Be Peering?

Google 貼牌冰箱（Google Refrigerator）

Google 貼牌冰箱（Google Refrigerator） (Photo credit: Aray Chen)

No, this is not an invitation for you to become peeping toms, dear readers. By peering I am talking about the process of trading Internet traffic directly with other networks to avoid paying to transport all of your Internet traffic to the major Internet POPs.

Peering didn’t always make a lot of sense, but there has been a major consolidation of web traffic to a few major players that has changed the game. In 2004 there were no major players on the web and internet traffic was distributed among tens of thousands of websites. By 2007 about 15,000 networks accounted for about half of all of the traffic on the Internet. But by 2009 Google took off and it was estimated that they accounted for about 6% of the web that year.

And Google has continued to grow. There were a number of industry experts that estimated at the beginning of this year that Google carried 25% to 30% of all of the traffic on the web. But on August 16 Google went down for about 5 minutes and we got a look at the real picture. A company called GoSquared Engineering tracks traffic on the web worldwide and when Google went down they saw an instant 40% drop in overall web traffic as evidenced by this graph: Google’s downtime caused a 40% drop in global traffic

And so, when Google went dead for a few minutes, they seem to have been carrying about 40% of the web traffic at the time. Of course, the percentage carried by Google varies by country and by time of day. For example, in the US a company called Sandvine that sells Internet tracking systems, estimates that NetFlix uses about 1/3 of the US Internet bandwidth between 9 P.M. and midnight in each time zone.

Regardless of the exact percentages, it is clear that a few networks have grabbed enormous amounts of web traffic. And this leads me to ask my clients if they should be peering? Should they be trying to hand traffic directly to Google, NetFlix or others to save money?

Most carriers have two major cost components to deliver their Internet traffic – transport and Internet port charges. Transport is just that, a fee that if often mileage based that pays for getting across somebody else’s fiber network to get to the Internet. The port charges are the fees that are charged at the Internet POP to deliver traffic into and out of the Internet. For smaller ISPs these two costs might be blended together in the price you pay to connect to the Internet. So the answer to the question is, anything that can produce a net lowering of one or both  of these charges is worth considering.

Following is a short list of ways that I see clients take advantage of peering arrangements to save money:

  • Peer to Yourself. This is almost too simple to mention, but not everybody does this. You should not be paying to send traffic to the Internet that goes between two of your own customers. This is sometimes a fairly significant amount of traffic, particularly if you are carrying a lot of gaming or have large businesses with multiple branches in your community.
  • Peer With Neighbors. It also makes sense sometime to peer with neighbors. These would be your competitors or somebody else who operates a large network in your community like a university. Again, there is often a lot of traffic generated locally because of local commerce. And the amount of traffic between students and a university can be significant.
  • Peering with the Big Data Users. And finally is the question of whether you should try to peer with Google, Netflix or other large users you can identify. There are several ways to peer with these types of companies:
    • Find a POP they are at. You might be able to find a Google POP or a data center somewhere that is closer than your Internet POP. You have to do the math to see if buying transport to Google or somebody else costs less than sending it on the usual path.
    • Peer at the Internet POP. The other way to peer is to go ahead and carry the traffic to the Internet POP, but once there, split your traffic and take traffic to somebody like Google directly to them rather than pay to send it through the Internet port. If Google is really 40% of your traffic, then this would reduce your port charges by as much as 40% and that would be offset by whatever charges there are to split and route the traffic to Google at the POP.

I don’t think you have to be a giant ISP any more to take advantage of peering. Certainly make sure you are peeling off traffic between your own customers and investigate local peering if you have a significant amount of local traffic. It just takes some investigation to see if you can do the more formal peering with companies like Google. It’s going to be mostly a matter of math if peering will save you money, but I know of a number of carriers who are making peering work to their advantage. So do the math.

Spying on our Internet Infrastructure

English: NSA EMPLOYEES ONLY Français : NSA emp...

English: NSA EMPLOYEES ONLY Français : NSA employés seulement (Photo credit: Wikipedia)

Everybody I know in the telecom industry has been following the controversy surrounding the allegations that the NSA has been gathering information on everybody’s Internet usage. What I find somewhat amusing are the smaller ISPs who are telling people that they have not cooperated with the NSA, and that it is ‘safe’ for customers to use them. That is a great marketing ploy but it far from the truth. The Internet infrastructure in the country is very complex, but for the most part the data traffic in the country can be characterized in three ways: core Internet, peering and private traffic.

The private data traffic is just that. There are huge numbers of private data connections in the country that are not part of the ‘Internet’. For example, every banking consortium has a private network that connects branches and ATMs. Large corporations have private connections between different locations within the company. Oil companies have private data circuits between the oil fields and headquarters. And for the most part the data on these networks is private. Most corporations that use private networks do so for security purposes and many of them encrypt their data.

The FBI has always been able to get a ‘wiretap’ on private data circuits using a set of rules called CALEA (Communications Assistance for Law Enforcement Act). The CALEA rules proscribe the processes for the FBI to use to wiretap any data connection. But over the years I have asked hundreds of network technicians if they have ever seen a CALEA request and from what I can see this is not a widely used tool. It would require active assistance from telecom companies to tap into private data circuits, and there just does not seem to be much of that going on. Of course, there is also not a lot of likelihood in finding spy-worthy information in data dumps between oil pumps and many of the other sorts of transactions that happen on private networks.

But the NSA is not being accused of spying on private corporate data. The allegations are that they are monitoring routine Internet traffic and that they possess records of every web site visited and every email that is being sent over the Internet. And it seems plausible to me that the NSA could arrange this.

The Internet in the US works on a hub and spoke infrastructure. There are major Internet hubs in Los Angeles, New York, Atlanta, Chicago, Dallas and Washington DC. Most of ‘Internet’ traffic ends up at one of these hubs. There are smaller regional hubs, but all of the Internet traffic that comes from Albuquerque, Phoenix, San Francisco, Las Vegas and all of the other cities in that region will end up eventually in Los Angeles. You will hear ISP technicians talking about ‘hops’, meaning how many different regional hubs an Internet transmission must go through before it gets to one of these Internet hubs.

So when some smaller Internet provider says that the NSA does not have their data they are being hopeful, naive or they are just doing PR. I recall an article a few months back where Comcast, Time Warner and Cox all said that they had not cooperated with the NSA and that it was safer to use their networks than to use AT&T and Verizon, who supposedly have cooperated. But everything that comes from the Comcast and Cox networks ends up at one of these Internet hubs. If the NSA has figured out a way to collect data at these hubs then there would be no reason for them to come to the cable companies and ask for direct access. They would already be gathering the data on the customers of these companies.

But then there is the third piece of the Internet, the peering network. Peering is the process of carriers handing data directly to each other rather than sending it out over the general Internet. Companies do this to save money. There is a significant cost to send information to and from the Internet. Generally an ISP has to buy transport, meaning the right to send information through somebody’s fiber cable. And they have to buy ‘ports’ into the Internet, meaning bandwidth connection from the companies that own the Internet portals in those major hubs. If an ISP has enough data that goes to Google, for example, and if they also have a convenient place to meet Google that costs less than going to their normal Internet hub, then they will hand that data traffic directly to Google and avoid paying for the Internet ports.

And peering is also done locally. It is typical for the large ISPs in large cities to hand each other Internet traffic that is heading towards each other’s network. Peering used to be something that was done by the really large ISPs, but I now have ISP clients with as few as 10,000 customers who can justify some peering arrangements to save money. I doubt that anybody but the biggest ISPs understand what percentage of traffic is delivered through peering versus directly through the more traditional Internet connections.

But the peering traffic is growing all of the time, and to some extent peering traffic can bypass NSA scrutiny at the Internet hubs. But it sounds like the NSA probably has gotten their hands on a lot of the peering traffic too. For instance, a lot of peering traffic goes to Google, and so if the NSA has an arrangement with Google then that catches a lot of the peering traffic.

There certainly are smaller peering arrangements that the NSA could not intercept without direct help from the carriers involved. For now that would be the only ‘safe’ traffic on the Internet. But if the NSA is at the Internet hubs and also has arrangements with the larger companies in the peering chains, then they are getting most of the Internet traffic in the country. There really are no ‘safe’ ISPs in the US – just those who haven’t had the NSA knocking on their doors.