More Details on Starlink

A few months ago Starlink, the satellite broadband company founded by Elon Musk, launched 60 broadband satellites. Since that launch, we’ve learned a few more things about the secretive venture.

We now know more details about the satellites. Each one weighs about 500 pounds. They are thin rectangular boxes like a flat-panel TV. Much of the surface is a solar panel, and each satellite also extends a second solar panel.

Each satellite comes with a krypton-powered ion thruster to use to navigate the satellite into initial orbit and to avoid future debris when necessary. This may sound like a cutting-edge propulsion system, but it’s been around for many years and the tiny engines create a small amount of thrust by shooting out charged ions of the noble gas – not a lot of thrust is needed to move a 500-pound satellite.

It seems the satellites can’t detect nearby space debris, so Starlink instead connects to the Air Force’s Combined Space Operations Center, which tracks the trajectories of all known space debris. The company will direct satellites to avoid known debris.

Probably the most important announcement for readers of this blog is that the company is likely to only compete in rural areas where there are few other broadband alternatives. This was finally admitted by Musk. There has been hopeful speculation in some parts of the industry that the low-orbit satellites would provide a broadband alternative everywhere, thus supplying a new competitor for cable companies. Since widespread competition generally results in lower prices there was hope that satellite broadband would act to make the whole broadband market more competitive.

We already had an inkling that satellite broadband was going to be rural-only when OneWeb, one of the competitors to Starlink, told the FCC that they were likely going to ultimately need about like 1 million wireless licenses for receivers. While that might sound like a huge number, one million satellite connections spread across the US is not creating a major competitor. We also heard the same message when several of the satellite companies talked about eventually having tens of millions of customers worldwide at maturity. Even with multiple satellite companies competing for customers there probably won’t be more than 3 – 4 million satellite broadband customers in the US – that would make a dent but wouldn’t fix the rural broadband gap. This strategy makes sense for the satellite companies since they’ll be able to charge a premium price for rural customers who have no broadband alternative instead of cutting prices to compete with cable companies.

There has still been no discussion from Starlink or the other competitors on broadband speeds or broadband pricing. It’s been nearly impossible to predict the impact of the satellites without understanding data speeds and total download capacity. The physics suggest that backhaul to the satellites will be the critical limiting factor, so it’s possible that there will be monthly data caps or some other way to control consumption.

One of the most interesting unanswered questions is how the satellites will do backhaul. Landline ISPs of any size today control cost and control data volumes by directly peering with the largest sources of broadband demand – being mostly Netflix, Google, Amazon, and Microsoft. As much as 70% of the traffic headed to an ISP is from this handful of destinations. Engineers are wondering how Starlink will handle peering. Will there be backhaul between satellites or will each satellite have a dedicated link to the ground for all data usage? This is a key question when a satellite is passing over a remote area – will it try to find a place within sight of the satellite to connect to the Internet or will data instead be passed between satellite with connections only at a major hub?

Answering that question is harder than might be imagined because these satellites are not stationary. Each satellite continuously orbits the earth and so a given customer will be handed off from one satellite to the next as satellites pass out of the visible horizon. The company says the receivers are about the size of a pizza box and they are not aimed at a given satellite, like what happens with satellite TV – instead, each receiver just has to be generally aimed skyward. It’s hard to think that there won’t be issues for homes living in heavy wooded areas.

One last interesting tidbit is that the satellites are visible to the naked eye. When the recent launch was first completed it was easy to spot the string of 60 satellites before they were dispersed. Astronomers are wondering what this will mean when there are ten thousand satellites filling the sky from the various providers. Elon Musk says he’s working to reduce albedo (the reflection of sunlight) to reduce any problems this might cause with land-based astronomy. But for stargazers this means there will always be multiple visible satellites crossing the sky.

The Consequences of Killing Network Neutrality

It looks almost certain that the FCC is going to kill Title II regulation, and with it net neutrality. Just as happened the last go around the FCC has already received millions of comments asking it to not kill net neutrality. And if you read all of the press you find dire predictions of the consequences that will result from the death of net neutrality. But as somebody who has a decent understanding of the way that broadband and the associated money flows in the industry I don’t think it will be as dire as critics predict, and I think there will also be unanticipated consequences.

Impact on Start-ups – the Cost of Access. One of the dire predictions is that a new start-up company that uses a lot of broadband – the next Netflix, Vine or Snapchat – won’t be able to gain the needed access with carriers, or that their access will be too expensive. Let me examine that conjecture:

  • Let me follow the flow of money that a start-up needs to spend to be on the web. Their direct largest cost is the cost of uploading their content onto the web through an ISP. The pricing for bulk access has always favored the bigger players and it’s more expensive today for a company that wants to upload a gigabyte per day compared to somebody that uploads a terabyte.
  • The normal web service doesn’t pay anything to then deliver their content to customers. Customers buy various speeds of download and use the product at will. Interestingly, it’s only the largest content providers that might run into issues without net neutrality. The big fights a few years ago on this issue were between Netflix and the largest ISPs. The Netflix volumes had grown so gigantic that the big ISPs wanted Netflix to somehow contribute to the big cost of electronics the ISPs were expending to distribute the service. The only way that there would be some cost to start-ups to terminate content would be if the ISPs somehow created some kind of access fee to get onto their network. But that sounds largely impractical. Bytes are bytes and they don’t exactly contain the name and billing address of the party that dumped the traffic on the web.
  • Some content like live video is a complicated web product. You can’t just dump it on the web at one location in the country and hope it maintains quality everywhere it ends up. There are already companies that act as the intermediary for streaming video to carry out the caching and other functions needed to maintain video quality. Even the big content providers like SlingTV don’t tackle this alone.
  • Finally, there will arise new vendors that will assist start-ups by aggregating their traffic with others. We already see that today with Amazon which is bundling the content of over 90 content providers on its video platform. The content providers benefit by taking advantage of the delivery mechanisms that Amazon has in place. This is obviously working and it’s hard to see how the end of net neutrality would stop somebody like Amazon from being a super-bundler. I think wholesalers like Amazon would fill the market gap for start-ups.

Paid Prioritization. The other big worry voiced by fans of Title II regulation is that it stops paid prioritization, or Internet fast lanes. There are both good and bad possible consequences of that.

  • It’s silly to pretend that we don’t already have significant paid prioritization – it’s called peering. The biggest content providers like Google, Netflix and Amazon have negotiated peering arrangements where they deliver traffic directly to ISPs in specific markets. The main benefits of this for the content providers is that it reduces latency and delay, but it also saves them from buying normal uploads into the open Internet. For example, instead of dumping content aimed at Comcast in Chicago onto the open web these big companies will directly deliver the Chicago-bound traffic to Comcast. These arrangements save money for both parties. And they are very much paid prioritization since smaller content providers have to instead route through the major Internet POPs.
  • On the customer side of the network, I can envision ISPs offering paid prioritization as a product to customers. Customer A may choose to have traffic for a medical monitoring company always get a priority, customer B might choose a gaming service and customer C might choose a VoIP connection. People have never had the option of choosing what broadband connections they value the most and I could see this being popular – if it really works.
  • And that leads into the last big concern. The big fear about paid prioritization is that any service that doesn’t have priority is going to suffer in quality. But will that really happen? I have a fairly good broadband connection at 60 Mbps. That connection can already deliver a lot of different things at the same time. Let’s say that Netflix decided to pay my ISP extra to get guaranteed priority to my house. That might improve my Netflix reception, although it already seems pretty good. But on my 60 Mbps connection would any other service really suffer if Netflix has priority? From what I understand about the routing of Internet traffic, any delays caused by such prioritization would be miniscule, probably in microseconds, which would be nearly imperceptible to me. I can already crash my Internet connection today if I try to download more content than it can handle at the same time. But as long as a customer isn’t doing that, I have a hard time seeing how prioritization will cause much problem – or even why somebody like Netflix would pay an ISP extra for it. They are already making sure they have a quality connection through peering and other network arrangements and I have a hard time understanding how anything at the customer end of the transaction would make much difference. This could be important for those on slow broadband connections – but their primary problem is lack of broadband speed and they are already easily overwhelmed by too much simultaneous traffic.

I am not as fearful of the end of net neutrality as many because I think the Internet operates differently than what people imagine. I truly have a hard time seeing how the ending net neutrality will really change the way I receive broadband at my home. However, I do have big concerns about the end of Title II regulation and fear things like data caps and of my ISP using my personal information. I think most of folks real concern is about Title II regulation, but that’s too esoteric for most folks and we all seem to be using the term ‘network neutrality’ as a substitute for that.

How Comcast Pays for Bandwidth

comcast-truck-cmcsa-cmcsk_largeToday’s blog is more about Comcast’s data caps. I recently saw a quote from Brian Roberts, the Comcast CEO during an interview by Business Insider. When asked about the data cap trials he said the following:

We don’t want anybody to ever not want to stay connected on our network, but just as with every other thing in your life, if you drive 100,000 miles or 1,000 miles, you buy more gasoline. If you turn on the air conditioning to 60 vs. 72, you consume more electricity. The same is true for usage, so I think the same for a wireless device. The more bits you use, the more you pay.

He is basically saying that it costs Comcast more to buy Internet bandwidth for customers who use more bandwidth. Certainly his first example means that – you certainly must buy more gas to drive a vehicle more miles. Is this a good analogy? For it to be true Comcast would have to be buying raw bandwidth each time a customer uses the Internet – this would mean when you download something at your house that Comcast is somehow buying more bandwidth from the big Internet spigot.

But that’s not how it works. While Comcast is really big, they are not one of the companies that owns the Internet, so they must buy bandwidth just like any other ISP. So how do ISPs buy Internet access? They buy it with two cost components – transport and raw bandwidth. Transport is the cost of getting the bandwidth from one of the major Internet POPs to a market. At Comcast’s size they either have a direct physical presence at each major Internet POP or they have an arrangement with some carrier who does. Due to their sheer size, I have to imagine that Comcast’s cost for transport on a per-megabit basis is lower than anybody else in the industry other than maybe AT&T, who is one of the owners of the Internet structure.

Transport can be a major cost for an ISP that operates a long distance from a major POP. I have small ISP clients that spend between $10,000 and $20,000 per month on transport, which is a lot if you only have a few thousand customers. But for Comcast this cost has to be miniscule on a per customer basis. And the cost is fixed. Once you buy transport to a market it doesn’t matter how much bandwidth you shove through the pipe. So this cost doesn’t increase due to customer usage.

The other cost is to buy the actual Internet connectivity — an expense that is sometimes referred to as an Internet port. This is an electronic connection directly into the main Internet routers. My small clients pay anywhere from $1 to $5 per raw dedicated megabit per month for Internet bandwidth. Generally the more you buy the cheaper it gets. Again, one has to imagine that Comcast pays a lot less than my clients due to their huge size.

And even that cost can be significantly reduced by large ISPs like Comcast through peering. Peering is where a carrier like Comcast makes a direct connection to companies with a lot of Internet usage like Netflix or Google. From an economic standpoint, peering is essentially the sam as transport and bypasses paying for the Internet port. Any traffic that goes through the peering connection does not increase with a customer’s use of the bandwidth.

An ISP’s total cost for an Internet port is based upon the average of the busiest times of the month. For instance, a small ISP might use 500 raw megabits of aggregate usage on most evenings, but if their customers have a few nights per month where they use 700 megabits, then the ISP pays for that larger amount for the whole month.

The interesting thing about this pricing structure is that the ISP pays the same every day of the month whether the customers are using the data or not. The cost to Comcast wouldn’t change if any one customer, or even all of the customers in a city, were to use more data, as long as that usage doesn’t create a new fastest day of the month. From a cost accounting basis, this means that the cost of Internet bandwidth can also be considered as a fixed cost. There is nothing that any one customer, or even a fairly large pile of customers, can do to change the cost of the bandwidth to Comcast. It does not cost them more when you watch an extra movie.

The idea that Comcast is paying more for somebody who downloads 500 gigabits per month than somebody who uses half of that is false. If that 500 gigabit customer was to instead use zero bandwidth in a given month then Comcast’s costs wouldn’t change by a penny.

To put this into a different perspective, many of my clients have done the math and in aggregate their bandwidth costs them between $2 and $5 per customer per month depending upon how small they are. This is the average of transport costs, any peering costs and the Internet port costs. I would be surprised if a large ISP like Comcast is spending much more than $1 to $2 per customer per month for bandwidth. For them to charge $35 for going over their data cap is outrageous and that charge is 100% profit to them.

Roberts did make one true analogy when he compared his data caps to wireless carriers like Verizon and AT&T. They buy bandwidth in the same way that Comcast does, and so it also doesn’t cost them extra when a customer uses an additional gigabit on their cellphone. US wireless data is very close to the most expensive bandwidth in the world and you have to go to places like Africa to see bandwidth being sold for as high pf a price. The cable companies like Comcast have eyed the Verizon and AT&T wireless profits with envy and the data caps are nothing more than an attempt to greatly bump up what they can bill for data. They will be billing customers for going above an arbitrary cap, while in reality using more bandwidth doesn’t cost Comcast anything extra.

The FCC and Peering

Zeus_peering_around_a_corner__(9386751334)As the politics of net neutrality keep heating up, Senator Pat Leahy and Representative Doris Matsui introduced the Online Competition and Consumer Choice Act of 2014.

This bill requires the FCC to forbid paid prioritization of data. But then, Senator Leahy was quoted in several media outlets talking about how the bill would stop things like the recent peering deal between Netflix and Comcast. I’ve read the proposed bill and it doesn’t seem to ban those kinds of peering arrangements. His comments point out that there is still a lot of confusion between paid prioritization (Internet fast lanes) and peering (interconnection between large carriers). The bill basically prohibits ISPs from creating internet fast lanes or in disadvantaging customers through commercial arrangements in the last mile

The recent deals between Netflix and Comcast, and Netflix and Verizon are examples of peering arrangements, and up to now the FCC has not found any fault with these kinds of arrangements. The FCC is currently reviewing a number of industry peering agreements as part of investigating the issue. These particular peering arrangements might look suspicious due to their timing during this net neutrality debate, but similar peering arrangements have been around since the advent of the Internet.

Peering first started as connection agreements between tier 1 providers. These are the companies that own most of the long haul fiber networks that comprise the Internet backbone. In this country that includes companies today like Level3, Cogent, Verizon and AT&T. And around the world it includes companies that you may not have heard of like TeliaSonera and Tata. The tier 1 providers carry the bulk of the Internet traffic and peering was necessary to create the Internet as these large carriers need to be connected to each other.

Most of the peering arrangements between the tier1 carriers have been transit-free or what is often referred to as bill-and-keep. The traffic between the major carriers tends to balance out in terms of originating and terminating volumes and in such cases it doesn’t make a lot of sense for two carriers to bill each other for swapping similar amounts of data traffic.

But over time there were peering arrangements made between the tier 1 carriers and tier 2 providers that includes the large ISPs and telcos. Peering was generally done in these cases to make the network more efficient. It makes more sense to interchange traffic between and ISP and somebody like Level3 at a few places rather than at hundreds of places. It’s always been typical for these kinds of peering arrangements to include a fee for the tier 2 carrier, something that is often referred to as a transit fee.

There is no industry standard arrangement for interconnection between tier 1 and tier 2 providers. And this is because tier 2 providers come in every configuration imaginable. Some of them own significant fiber assets of their own. Others, like Netflix have a mountain of one-directional content and own almost zero network. And so tier 2 providers scramble to find the best commercial arrangement they can in the marketplace. One thing that is almost universal is that tier 2 providers pay something to connect to the Internet. There is no standard level of payment and transit is a very fluid market. But payment generally recognizes the relative level of mutual benefit. If the traffic between two parties is balanced then the payments might be small or even free. If one party causes a lot of costs for the other then payments typically reflect that imbalance.

Netflix has complained about paying Comcast and Verizon. But those ISPs wanted payments from Netflix since the traffic from Netflix is large and totally one-directional. Comcast or Verizon needs to construct a lot of facilities in order to accept the Netflix traffic and they don’t get any offsetting benefit of being able to send traffic back to Netflix on the same connection.

In economic terms, on a national scale the peering market is referred to as an n-dimensional market, meaning that a large tier 2 provider has the ability to negotiate with multiple parties to achieve the same result. For example, Verizon has a lot of options for moving data from the east to the west coast. But eventually the Internet becomes local, and that is where the cost and the contention arises. As Internet traffic enters a local metropolitan market it begins to hit choke points where the traffic can overwhelm the local facilities and cause congestion. The payments that Comcast or Verizon want from Netflix are to build the facilities needed for getting Netflix movie traffic to and through these local hubs and chokepoints.

Peering arrangements like this make sense. I find it hard to believe that the FCC is going to get too deeply involved in peering arrangements. It’s an incredibly dynamic market and carriers are constantly rearranging the network as they find better prices or more efficient network arrangements. If there is any one place where the market works it is between the handful of large carriers that handle the majority of the Internet traffic. Most of the bad things that can happen to customers are going to happen in the last mile network, and that is where net neutrality should properly be focused.

And why the picture of the kitten? I work at home and at my very local part of the network this is the kind of peering that I often get.

Is There a Web Video Crisis – Part III

Zeus_peering_around_a_corner__(9386751334)In the earlier two installments of this blog I looked at various components of the Internet backbone to see if any of them might be the reason why Comcast and Verizon want to charge NetFlix and other content providers for and Internet ‘fast lane’. I’ve looked at the fiber and distribution networks as well as the routers in data centers. I also considered caching. In this article I will finally look at peering as a possible reason why there ought to be an Internet fast lane.

Peering is when two networks directly interchange traffic rather than let it route over the open Internet. Peering is done to save money and the savings can be significant. A company like Comcast pays something less than a dollar per dedicated megabit to accept traffic from the Internet. If they can instead have Google hand them the traffic coming from Google services they can avoid paying for the bandwidth that traffic would have incurred coming through the open Internet.

I would assume that Comcast and Google peer because Google peers with many of my much smaller clients. Peering involves having a fiber cross-connection between the two companies at a data center in each Comcast market. Each party would normally be responsible for their own routers and collocation costs at a data center. So Comcast’s cost of peering with Google are relatively small charges for collocation and cross-connection in the data center, while the savings would be gigantic.

This is a good place to note the difference to Comcast for traffic they receive from the web and traffic they send to the web. The companies that sell Internet access sell symmetrical data pipes that provide the same amount of bandwidth in both directions. But Comcast does not sell symmetrical data product to their customers and they provide vastly faster download speeds than upload speeds. For example, the Comcast 100 Mbps download product only has an upload speed of 5 – 6 Mbps. This means that the real cost to Comcast and similar ISPs for Internet traffic is paying for downloading because they have a huge amount of excess capacity in the upload direction. Peering saves them so much money because it shrinks the size of download pipe they must purchase.

So it’s a given that Comcast saves money by peering with Google. With peering they would not have to purchase the bandwidth to provide all of the accumulated traffic for Gmail, Google Search, Google Maps, all the android apps being used on home WiFi networks. It is estimated that for most ISPs that Google is involved with around 25% of all web traffic, so peering with Google can save Comcast from buying a significant amount of Internet bandwidth.

It also costs Google money to peer with Comcast because they also have to pay to use the data centers. But Google likes peering because it speeds up their traffic and gives their customers a better experience using Google products. Peering avoids the extra hops that come from using the open Internet. Generally both parties in a peering arrangement see it as a win:win situation.

Comcast is claiming that one of the reasons they need to charge for a ‘fast lane’ is to cover the costs of peering with NetFlix. I find that claim to be interesting. There is one subtle difference between Comcast’s traffic from Google and traffic from NetFlix. The traffic from NetFlix is one directional in the direction of Comcast while there is some traffic in both directions between Comcast and Google (although it is still mostly towards Comcast). This means that the peering savings for Comcast to peer with NetFlix is even more dramatic than it is with Google. Comcast saves so much money by peering with NetFlix that they could pay NetFlix to peer and still save a ton of money.

When the story about Comcast and NetFlix first came out it was somewhat confusing because NetFlix was using Level3 and other intermediate carriers between themselves and Comcast. It makes sense that they would do this because NetFlix doesn’t own any actual network. The presence of an intermediate carrier does not change the fact that peering with NetFlix is an incredibly good deal for Comcast. The press reports were confusing and it sounded like Comcast wants NetFlix to peer directly with them and not use intermediate carriers. I can only interpret that to mean that Comcast wants NetFlix to buy transport from them and not from intermediate carriers. And this might be how Comcast is ‘charging’ for the peering arrangement. What I find totally mysterious in all of this is how Comcast is using the peering arrangement as a reason why they should be able to charge anything to NetFlix. Again, Comcast saves so much money through this peering that they ought to be the ones paying NetFlix to peer. The whole peering argument has me scratching my head.

And the picture of a cat? It’s peering!

Is Peering the End of Network Neutrality?

Network_neutrality_poster_symbolNetFlix and Comcast have announced a deal whereby NetFlix will pay to peer into the Comcast network. Numerous articles popped up yesterday talking about how this is the end of network neutrality. But I am not so sure about that. In order to understand this, let me talk a bit about how peering works today. Peering is when two networks decide to make a direct connection between the networks rather than connecting in a more traditional way through the open Internet.

There are two kinds of connections that are typically made. One is local peering. This is when two networks who are geographically close decide to exchange data traffic. This typically benefits both parties. Let’s look at an example of why. Let’s assume the two parties are medium sized carriers, one a telephone company and the other a cable company that are competing in the same community. There is always a considerable amount of Internet traffic that is conducted within a community. People browse the websites of stores in their own community. People do on-line banking with local banks. People work at home and want to get data into and out of their employer’s local networks.

Normally each of these carriers would deliver traffic between their two networks, say between a customer on one network and a bank on the other one by sending this traffic to the open Internet. Each company will have a connection to the Internet, through some wholesale provider that will terminate eventually at one of the major Internet pops like Chicago or Dallas. And so when a customer wants to connect with his bank, the data will travel out through the first network to the major pop where it will be handed off to the data stream going back to the second network.

Such a connection is said to make at least several hops, meaning the times that the message is handled by a data router somewhere in the network in order to figure out where it is going. The more hops, the slower the connection. But local peering solves this problem because the traffic can be exchanged locally and goes straight from one carrier to the other without being sent first to some distant POP. This is a simplistic description because peering arrangements are usually more complicated than this. They are more likely to be between the underlying transport carriers that handles the traffic for the telephone company and the cable company. But peering will make the connection more direct than it would be under normal network circumstances.

The other kind of peering is one that saves money. I have many clients who peer with Google because Google and all of its various subsidiaries accounts for a significant percentage of the traffic on any Internet connection. My clients have done the math and see that it is cheaper to make a direct connection with Google rather than paying their underlying carrier to get it to Google. Anybody who peers with Google this way must pay out of their own pockets to get to a Google POP, probably including paying for the equipment at the POP needed to make the connection. But this kind of peering often results in a significant savings. Most people’s connection with Google is very much one-directional. There is usually a lot more traffic coming from Google than going to Google.

We don’t have the details of the Comcast / NetFlix deal to be certain what the arrangement is. But up until now it’s clear that the two sides have not agreed to a direct peering arrangement. One has to assume that the connection from NetFlix is nearly all in one direction – to download video to customers who sit on the Comcast network. Without a direct peering arrangement the traffic must get to Comcast through intermediate carriers and often would be routed in ways that would slow up the traffic, as is any traffic on the open Internet.

I would assume that there is not one big Comcast network, but instead there are pockets of Comcast all over the country. I would assume that for NetFlix to fully peer with Comcast that they are going to have to make connections with these various pockets, all at NetFlix cost. And if this was normal peering, NetFlix would also be expected to pay for the connections into the Comcast network including owning or somehow paying for the large amount of equipment needed to terminate their traffic.

Again, the two sides aren’t talking about the details. But I would expect it to cost NetFlix something to get their traffic directly to all parts of the Comcast network. That is how normal peering works. Where the line of network neutrality will have been crossed is if NetFlix has to pay a lot more for this connection than what others pay. But since this deal has been under negotiations for a year, one has to assume that both parties had the old network neutrality rules in mine as it was negotiated. I can certainly envision an arrangement that is more like normal peering than of a big violation of the principles of network neutrality. If it was the latter I would expect NetFlix to be putting up a big stink. Network neutrality benefits companies like NetFlix tremendously, and if they aren’t complaining then there is a good chance that this is peering like normal and not a giant money grab by Comcast.