The San Francisco Broadband Experiment

The City of San Francisco seems poised to tackle building fiber to everybody in the city. They have conducted several studies looking at the cost of building fiber. The city also created a Blue Ribbon Panel that has recommended in this recent report that the city should construct a fiber network.

The city is proposing to finance a fiber network in a new way. The city is looking at fiber connections in the same way as any other utility like electricity or water. The concept is that everybody in the city would pay a monthly utility fee that would fund the construction and operation of a fiber network. The number that was tossed around earlier this year was an average monthly fee of $26 per month to be charged to every household and business in the city. It’s hard to tell from the various reports if that’s the number still being considered. The Blue Ribbon report does recommend that the city seek private investment which would be used to lower that number.

The city wants to build the fiber network to everybody in the city, which differs from the typical ISP demand model that only builds to those that buy a broadband connection. The city does not want to be an ISP and wants to emulate some of the large cities in Europe which open up their fiber networks to multiple ISPs. The hope is that multiple ISPs using the network for a minimal charge will create competition and low-price broadband.

It’s an interesting concept. There are smaller municipalities in the country that are financing fiber with municipal bonds – but in most cases the expectation is that the fiber project will generate enough revenue to repay the bonds. But fiber construction is expensive in big cities and the utility fee is needed to finance a network that will cost more than $1 billion in San Francisco.

The city’s rationale for considering this is to provide world-class broadband to everybody. This is a city that is in direct economic competition with cities in Japan, Taiwan and South Korea – and the city views fiber as a necessary component to long-term financial success. Comcast is the biggest ISP in the city and they have fast broadband today with speeds now up to a gigabit download. AT&T offers DSL plus has built fiber to large businesses and MDUs. Sonic has been building some fiber to residences in the Bay Area. And like in every large city there has been some fiber built by ISPs and CLECs to selected locations in the city.

But the city is concerned that a significant percentage of the public can’t afford fast broadband access today. The Blue Ribbon Panel notes that the government-sponsored fiber network in Singapore reduced broadband prices from $90 per month down to $30 – $40 today while speeds leaped to a symmetrical gigabit connection.

No NFL city has yet tried to build fiber and this proposal is going to meet a lot of resistance. Certainly Comcast, AT&T and other big ISPs will do everything possible to derail such an effort. The city says that they don’t want to directly compete with commercial ISPs, but if the fiber network really lowers gigabit prices to $30 – $40 that will clearly get most of the customers in the city.

I foresee all sorts of attempts to try to stop this project. The big ISPs are enjoying unprecedented support today in Congress and the FCC, and one ISP tactic might be to legislate against the project – either at the federal or state level. My fear is that a legislative approach might also stop more traditional municipal broadband projects. I would also expect to see numerous lawsuits from ISPs challenging the project. It’s such a new concept that it’s hard to envision the basis for such lawsuits, but I fully expect them. I can also envision a few citizen lawsuits trying to stop a mandatory new utility fee – picture forcing Comcast employees to pay to construct a competing network.

The final big hurdle will be in getting enough quality ISPs on the network to offer real customer choice. The few open access networks in this country have not attracted the many quality ISPs. The open access model works in Europe because the old state-monopoly telcos and cable companies have been forced into competing with each by the formation of the European Union. And perhaps quality ISPs will take a chance on a network in an NFL city. But in this country there seems to be agreement among cable companies to not compete with each other and it’s unlikely that we would see Charter, Mediacom or others stepping in to compete against Comcast.

This is a really interesting idea and it could be a viable way to get gigabit broadband to everybody in a big city. The city has not made the decision to take the leap forward, and if they do they will certainly face an uphill battle to make it work. But this could be the first trial in trying to bring the European open access model to the US.

Broadband Access to Apartment Buildings

Seal_of_San_FranciscoMark Ferrel of the Board of Supervisors for San Francisco has proposed an ordinance that would require multi-tenant buildings to provide access to broadband providers. This would apply to both residential and commercial properties.

You can understand why the city wants to tackle this issue. The nationwide percentage of families living in apartments is around 35%, but in San Francisco apartments represent 63% of housing units. And the percentage of families living in apartments is high in a lot of big cities – in New York City it’s 68%; in Seattle it’s 54%; in Atlanta it’s 56%.

Cities understand that bringing fiber to their city is not enough if it only benefits single-family homes and standalone businesses. As I wrote in a recent blog there are millions of urban households that don’t have access to broadband, and a lot of these situations are due to apartment owners that have excluded broadband providers. It’s fairly normal for apartment owners to have made deals years ago with service providers to serve their buildings on a revenue-sharing basis. The landlord may or may not include the triple-play products in the rent, but they get a kick-back from the service providers as a reward for exclusive access.

A few years back the FCC put some restrictions on cable companies and ISPs from entering into certain kinds of exclusive arrangements with building owners. It was a fairly common practice, for example, for an ISP to agree to wire a building for free, but then retain ownership of all of the wiring. In these cases the owner gave up all rights to the cable company or telco and it’s those entities that keep out competition.

I think there is a general impression that the FCC order forbade the cable companies from entering into all exclusive arrangements. But unfortunately it did not and it instead bans only certain types of arrangements – but not all. So I would expect at some point for this ordinance to be challenged at the FCC when it bumps into arrangements that are still allowed. But I think cities expect legal challenges when they tackle new ground.

This is also not going to be as clean-cut as Mr. Ferrel is hoping for. The ordinance grants access to multi-tenant buildings to any state-licensed ISP. Unfortunately, in many buildings there are physical restrictions to allowing even a second ISP. There might be very limited space for an ISP to put a rack of equipment. There are often issues with having enough space in the risers (the conduits that carry wires between floors).

And many apartments can’t accommodate having ‘open access’ where any ISP can gain access to the wiring for any unit from some central location. Unfortunately many apartments are not wired with ‘home run’ drops that go from a core to each unit, but instead often share the same cabling for multiple units.

There are often limited options for getting new wires to apartments. I can picture some really messy situations if multiple ISPs are granted access to the same buildings and each tries to string cables through hallways or other public areas. You can picture the same sort of clutter that we often see on urban poles with too many wires crammed into limited space.

But even with all of these issues, Mr. Ferrel is on the right track. The fact is that many apartment dwellers are being denied access to fast Internet due to arrangements made by the building owners. This ordinance is the first attempt I’ve seen for solving the lack of broadband and choice for a large percentage of urban households.

The ordinance tries to be fair to apartment owners and allows them to expect reasonable compensation for access to their buildings. Obviously that concept will need work to put into practice, but the ordinance doesn’t open up buildings to anybody to build without rules.

There is one thing the ordinance doesn’t tackle. What if nobody wants access to an apartment? A significant portion of urban apartments without good broadband access are low-income housing and ISPs and incumbents have been accused of redlining such customers for years. So this kind of ordinance can’t solve everything – but it’s a start.

A New Model for Open Access?

Fiber CableThe traditional open access business model to serve residential customers has never worked in this country. I am familiar with the financial performance of most of the existing open access networks and from a purely financial perspective they are all failures. A few networks have failed outright like Provo. A few others have been able to generate enough revenues to cover annual operating costs, but most don’t even do that. And from what I’ve seen, none of the existing open access networks have ever been able to generate enough cash to pay anything towards the cost of building the fiber network, leaving the cities that build the network holding the financial bag for the initial investment.

There are a few reasons that this has never worked. First is that open access naturally drives ISPs towards cherry picking. Open access networks operate by charging fees to ISPs to use the network. If an ISP pays the typical $30 per month fee to use the network, they are not going to sell inexpensive broadband to anybody in the community. So when ISPs only sell high-priced products they don’t get enough customers and the city network owner doesn’t collect enough revenue to pay for the network. This has happened to every traditional open access network. None of them have signed up enough customers to pay for the networks and everyone who has built a network using this model ends up heavily subsidizing the open access network.

The other issue is that most cities have had trouble attracting very many quality ISPs. The whole concept of open access is to offer choices to customers. But most of the open access networks in the country only have a few ISPs, and even the ones they attract are often tiny, undercapitalized businesses. Attracting ISPs is so hard that there is one large open access network today that has been reduced over time to having only one residential ISP on the network. That’s not providing much customer choice.

But there are two cities looking at an alternative model. One is the small city of Ammon, Idaho, and the other is San Francisco. Both cities want residents to pay for the basic cost of the network. It’s an interesting idea.

In Ammon a household that wants broadband access will pay a tax levy of $10 to $15 per month and will also pay a utility fee of $16.50 per month. This means that each subscriber will pay $26 to $31 per month for the fiber network – a very similar charge to what is charged to ISPs on other open access networks. The Ammon commitment is voluntary and only those that sign up for broadband will pay the fees.

San Francisco is considering a similar proposal. There, residents would pay a monthly utility fee of $25 and businesses would pay as much as $115. In San Francisco this fee would be mandatory and everybody in the City would be assessed the fee. In an NFL city the fee probably has to be mandatory to assure that the network will be paid for.

Having customers pay a fee to the city takes the pressure off the cherry picking issue. By lowering what ISPs pay there is a lot better chance of having affordable products on the network. And that ought to result in more customers on the network.

But like any idea this one still leaves some open questions. For instance, how does the city make enough money over time to pay for the inevitable replacement of electronics or catastrophic events like storm damage? Or what does a city do if the ISPs don’t do a good job and customers don’t like them? The Ammon plan requires the payment of fees for a very long time, and small businesses like ISPs often don’t have the staying power to last for a long time. How will the business keep up with inflation – will the fees have to increase every year? And what happens if the city doesn’t get or keep enough customers to pay for this – will the fees go up for everybody else or will the city subsidize the network?

In a voluntary system like Ammon I also wonder what the consequences are for homes that change their mind over time. What if somebody has a financial problem and is unable to pay the fees? What happens when they want to sell their home – is this fee a tax lien of sorts? That’s what has happened to homes that buy solar power systems that are paid for over time. And what happens if a new buyer doesn’t want the fiber and doesn’t want to pay the fee? No doubt over time there will be legal issues to figure out.

The challenge to make this work in San Francisco seems much more difficult. It’s not hard to envision lawsuits from citizens who don’t want to pay the fees. And I can imagine a fierce battle with Comcast and the other current ISPs over the legality of a mandatory fiber utility fee. This seems like a concept that could take a decade of court time to resolve.

But the idea of having citizens somehow pay for the fiber network is an interesting one. Irrevocable customer pledges are a revenue stream that can be used to finance fiber construction. It’s hard to know if this concept will work until we see it in action. But it shows how serious cities are becoming to get good broadband. One has to think that if households are willing to sign long-term pledges to pay for fiber that it has to make a difference. I am sure communities all over the country will be watching to see if this works.

Another Municipal Model

Seal_of_San_FranciscoCity officials in San Francisco recently issued a report that takes a very different stance than most other cities that are looking at broadband issues. The city essentially rejects the normal demand-based commercial model for broadband and looks at a new structure that will bring broadband to everybody.

The report is authored by the office of supervisor Mark Farrell and reflects some of the recommendations from the San Francisco Municipal Fiber Advisory Panel. The report very correctly observes that today’s commercial broadband model leaves a lot of citizens without broadband. Numerous nationwide surveys have shown that the majority of households without broadband access today feel they cannot afford the market prices for service.

So the San Francisco report recommends that the City institute a $26 per month fee on all households – with a higher fee on businesses – to help pay for broadband to everyone. They further recommend a public private partnership model to operate the business and assume that tiered pricing will still allow for profitability for a commercial partner.

The numbers are based upon an estimate that it would cost $867.3 million to build a fiber network in the city and $231.7 million per year to maintain the network. In my experience in looking at other large cities, both numbers feel very high. One has to assume in an open access network where fiber was built to everybody that the ongoing maintenance expenses for a network would be far lower than that since much of those costs would accrue to the ISPs and not to the city.

The city is not the first place that has looked at paying for fiber using taxpayer money, but they are by far the largest. A few small communities like Leverett, Massachusetts have paid for fiber construction with tax levies. The city of North Kansas City built a network and essentially is giving free service to residents for the next ten years. And the Utopia system in Utah recently looked at the tax-payer funding model, although it looks like a lot of the communities involved are rejecting the idea.

It’s a very interesting concept that has a bunch of pros and cons. On the plus side this would certainly solve the digital divide if every household in a community had a fiber connection. There would still be the issue to solve of making sure that everybody has a computer, but that seems like an easier problem to solve than getting the fiber network built to everybody.

But I can foresee a few major hurdles in implementing such an idea in an NFL city, such as the following:

  • The City probably doesn’t have the right to insist that they can bring fiber into apartment buildings. The FCC has made it clear that building owners have the right to control the wiring and the access to services on their own property. Many of the apartment owners will already have made a long-term contractual arrangement and be doing revenue sharing with the local cable company or some other service provider.
  • One can envision multiple lawsuits from citizens and businesses that wouldn’t want the city solution or who won’t want to pay the fee. It’s one thing to do this in a tiny town like Leverett, MA where there was no existing broadband, but in a large city there are bound to be many who don’t want the city doing this.
  • This is such a drastic solution that it surely would invite legislative action and multiple lawsuits from the incumbent providers. California is one of the states that allows for municipal competition, but using direct tax revenues to compete against the existing broadband providers would raise legitimate concerns about unfair competition. One can envision attempts to pass state or national legislation that would outlaw the proposed business plan. ISPs would use every tool at their disposal to fight this for fear that it might work and could spread elsewhere.

As the report points out, cities have a broadband dilemma today. Even where there is fiber or good broadband today there are a lot of households that can’t afford broadband. The report estimates there are over 100,000 people in San Francisco that can’t afford the market price for broadband and another 50,000 that still use dial-up.

There is also the issue of carriers building to just some parts of a city. One only has to look at all of the east coast cities that have Verizon FiOS to see the result of allowing commercial broadband providers to cherry-pick in markets. These cities have some neighborhoods with fast fiber broadband and competition between the telco and the cable company (which many observe is not real competition). But they have many neighborhoods without fiber and none of these cities can formulate a business plan that can justify bringing fiber to the neighborhoods that Verizon bypassed as too expensive to build.

The San Francisco report was a little fuzzy on a few of the details, which is natural since those details can only be made clear through negotiations with carriers willing to operate on such a network. You have to give the city kudos for creativity. But I foresee a big uphill battle if they try to implement this. But it’s an idea that should work if it can overcome the opponents that will spend huge money to try to prevent it.

The Quiet Expansion of Wi-Fi Networks

Wi-Fi

Wi-Fi (Photo credit: kristinmarshall)

I am sure I am like most business travelers and one of the first things I look for when I get to a new place is a WiFi connection for both my laptop and cellphone. Finding WiFi lets me get online with the computer and stops me from racking up data charges on my cell plan.

And for the longest time there has been very little public WiFi outside of Starbucks and hotels. But that is starting to change, at least in some places. There are several companies that have quietly been pursuing w WiFi deployments.

The biggest of these is the cable companies. It’s hard to get accurate counts of how many hot spots they have deployed. In 2012 a consortium of cable companies  – Comcast, Cox, Time Warner, Bright House and Optimum – banded together as the Cable WiFi consortium to deploy hotspots. Comcast claims that the industry has deployed over 300,000 hot spots. However, the Cable WiFi web site claims over 200,000. But whatever the number this is far larger than anybody else.

The Cable WiFi networks are offered to the customers of those companies as a mobile data extension of their service. Today these hotspots are centered around big cities – the northeastern corridor, San Francisco, Chicago, Los Angeles, Tampa, Austin and others.

The next biggest provider is AT&T which claims about 30,000 hot spots. AT&T claims over 705 million WiFi connections onto its WiFi network in the fourth quarter of 2012. However, Google has announced that it is getting in the game and nobody knows how big they might get with this effort. But their first announcement is that they are taking over all of the hotspots at Starbucks Coffee (which is a lot of the AT&T hotspots).

The cable companies have been deploying the hotspots in several ways. In some communities they are installing them on utility poles. In other situations they are going into establishments similar to the Starbucks WiFi.

WiFi is becoming more and more important to people’s daily life, so this trend is going to be very popular. Cellphone plans are getting stingier and stingier with cellular data at the same time that cell phones and tablets have the ability to use more and more data. If that data is not offloaded onto WiFi networks then customers are facing some gigantic cellphone bills.

WiFi is never going to be a replacement of cellular. For example, the technology used and the spectrum used make it very difficult to do dynamic handoffs like happens with your cell phone. You can literally walk out of WiFi coverage on foot where cellular coverage will stick with you driving at speeds of 60 miles per hour.

But people are finding more and more uses for WiFi all of the time, and so the desire for public WiFi is probably going to explode. The cable companies report that every time they open a new hot spot that usage explodes soon after people figure out it is available. One area where they have seen the biggest use is at the Jersey shore where vacationers and visitors are relieved to find WiFi available.

Anybody building a fiber network ought to consider a wireless deployment. There are several ways to monetize the investment. The obvious revenue from WiFi is through daily, weekly and monthly usage fees. But if you are a triple play provider, a more subtle benefit of wireless is in making your customers stickier since you are giving them a mobile component of their data service. Another revenue stream is to sell prioritized WiFi access to the local municipality, electric company and others, with priority meaning that their employees get a prioritized access to the network, with first responders trumping everybody else. There are also smaller revenue streams such as earning commissions on the DNS traffic for people who purchase products over your WiFi network.

How Vulnerable is the Internet?

OLPC: XO internet access

OLPC: XO internet access (Photo credit: Wikipedia)

A question you hear from time to time is how vulnerable the US Internet backbone is in terms of losing access if something happens to the major hubs. The architecture of the Internet has grown in response to the way that carriers have decided to connect to each other and there has never been any master plan for the best way to design the backbone infrastructure.

The Internet in this country is basically a series of hubs with spokes. There are a handful of large cities with major regional Internet hubs like Los Angeles, New York, Chicago, Dallas, Atlanta, and Northern Virginia. And then there are a few dozen smaller regional hubs, still in fairly large cities like Minneapolis, Seattle, San Francisco, etc.

Back in 2002 some scientists at Ohio State studied the structure of the Internet at the time and said that crippling the major hubs would have essentially crippled the Internet. At that time almost all Internet traffic in the country routed through the major hubs, and crippling a few of them would have wiped out a lot of the Internet.

Later in 2007 scientists at MIT looked at the web again and they estimated that taking out the major hubs would wipe out about 70% of the US Internet traffic, but that peering would allow about 33% of the traffic to keep working. And at that time peering was somewhat new.

Since then there is a lot more peering, but one has to ask if the Internet is any safer from catastrophic outage as it was in 2007? One thing to consider is that a lot of the peering happens today at the major Internet hubs. In those locations the various carriers hand traffic between each other rather than paying fees to send the traffic through an ‘Internet Port’, which is nothing more than a point where some carrier will determine the best routing of the traffic for you.

And so peering at the major Internet hubs is great way to save money, but it doesn’t really change the way the Internet traffic is routed. My clients are smaller ISPs, and I can tell you how they decide to route Internet traffic. The smaller ones find a carrier who will transport it to one of the major Internet hubs. The larger ones can afford diversity, and so they find carriers who can carry the traffic to two different major Internet hubs. But by and large every bit of traffic from my clients goes to and through the handful of major Internet hubs.

And this makes economic sense. The original hubs grew in importance because that is where the major carriers at the time, companies like MCI and Qwest already had switching hubs. And because transport is expensive, every regional ISP sent their growing internet traffic to the big hubs because that was the cheapest solution.

If anything, there might be more traffic routed through the major hubs today than there was in 2007. Every large fiber backbone and transport provider has arranged their transport networks to get traffic to these locations.

In each region of the country my clients are completely reliant on the Internet hubs. If a hub like the one in Dallas or Atlanta went down for some reason, ISPs that send traffic to that location would be completely isolated and cut off from the world.

There was a recent report in the Washington Post that said that the NSA had agents working at only a handful of major US Internet pops because that gave them access to most of the Internet traffic in the US. That seems to reinforce the idea that the major Internet hubs in the country have grown in importance.

In theory the Internet is a disaggregated, decentralized system and if traffic can’t go the normal way, then it finds another path to take. But this idea only works assuming that ISPs can get traffic to the Internet in the first place. A disaster that takes out one of the major Internet hubs would isolate a lot of towns from the region around it from having any Internet access. Terrorist attacks that take out more than one hub would wipe out a lot more places.

Unfortunately there is no grand architect behind the Internet that is looking at these issues because no one company has any claim to deciding how the Internet workd. Instead the carriers involved have all migrated to the handful of locations where it is most economical to interconnect with each other. I sure hope, at least, that somebody has figured out how to make those hub locations as safe as possible.