Operating on a Leased Network

One of the comments posted on a recent blog mentioned that CenturyLink recently had agreed to operate on somebody else’s fiber network to serve residential customers – the first time that one of the big telcos or cable companies had agreed to do so. One of the major reasons cited for lack of competition in the US is the unwillingness of the major ISPs to operate outside their own networks. This certainly sounded newsworthy and I looked into the example cited.

CenturyLink has agreed to use the fiber network provided by Lumiere Fiber, an affiliate company of Sterling Ranch, a new planned community outside of Denver. CenturyLink won the ability to serve the community through an RFP competition with Comcast, the cable company serving the area. As the winner, CenturyLink will be the exclusive ISP on the network – which only has a few homes now but has plans to grow to 12,000 residences.

So is this really newsworthy? I think the answer is both yes and no – but mostly no. It is true that CenturyLink will be using somebody else’s fiber network, and a large one at that, when the community is ultimately built. But there are a number of reasons why this is not as groundbreaking as it sounds.

First, this is not really unique. While this is a large new subdivision, in many ways this is similar to the thousands of arrangements that ISPs routinely have made to serve large apartment complexes. In the vast majority of apartments the wiring is owned by the landlord and not the ISPs. There are some large apartment units around the country numbering in the thousands of units and this opportunity is unique only from this perspective of being larger than most MDUS.

CenturyLink is already building a lot of fiber to residential neighborhoods, with nearly 1 million new units passed this year – so this isn’t going to present any technological challenges. I am sure that the company will use the identical electronics and provisioning software it uses everywhere else.

This also is not going to stretch the operational systems of CenturyLink. The only real difference between this and other CenturyLink fiber is that the company doesn’t own the fiber. But they are going to take orders and connect new customers using their normal processes. They will dispatch technicians for trouble calls in the usual manner. And if Lumiere hires CenturyLink to do the fiber maintenance then they would even make fiber repairs in almost the same manner (this detail was not specified in the press releases).

There seems to be two reasons why the big ISPs don’t generally use networks owned by others. In the case of the big cable companies there seems be a gentleman’s agreement to never cross those lines. I can’t find one example of a big cable company crossing the line to compete for residential customers.

But the hardest barrier for the big ISPs to use other networks is the fact that their systems are largely incapable of making operational exceptions. They have created operation systems and processes that work for them, on their own networks, with their own employees. These processes are often highly decentralized and it takes employees scattered across the country to accomplish normal daily tasks like adding a new customer or answering a trouble call. It’s extremely difficult for a decentralized company to make exceptions for customers that are treated different than everybody else – that always results in chaos.

An example of this is Verizon FiOS. When the company decided to build fiber they realized that they could not reshape their existing copper work processes and people to accommodate the new technology. They solved this by creating a totally new company and FiOS was new from top to bottom – from technology, to people, to processes.

The real headline I want to see is where one of the big ISPs gets on somebody else’s network in a competitive environment. For example, there are a number of open access fiber networks in Washington state that are significantly larger than the Sterling Ranch opportunity. There are numerous smaller open access networks around the country, and no big ISP has ever served residents on these networks. If the big companies would jump on competitive networks then a lot more of these networks would get built.

San Francisco is talking about building an open access fiber network and if it’s built will really challenge the big ISPs. If that network comes to fruition, will one of the other big cable companies decide to take on Comcast? That would be the big news we’ve always wanted to hear.

The San Francisco Broadband Experiment

The City of San Francisco seems poised to tackle building fiber to everybody in the city. They have conducted several studies looking at the cost of building fiber. The city also created a Blue Ribbon Panel that has recommended in this recent report that the city should construct a fiber network.

The city is proposing to finance a fiber network in a new way. The city is looking at fiber connections in the same way as any other utility like electricity or water. The concept is that everybody in the city would pay a monthly utility fee that would fund the construction and operation of a fiber network. The number that was tossed around earlier this year was an average monthly fee of $26 per month to be charged to every household and business in the city. It’s hard to tell from the various reports if that’s the number still being considered. The Blue Ribbon report does recommend that the city seek private investment which would be used to lower that number.

The city wants to build the fiber network to everybody in the city, which differs from the typical ISP demand model that only builds to those that buy a broadband connection. The city does not want to be an ISP and wants to emulate some of the large cities in Europe which open up their fiber networks to multiple ISPs. The hope is that multiple ISPs using the network for a minimal charge will create competition and low-price broadband.

It’s an interesting concept. There are smaller municipalities in the country that are financing fiber with municipal bonds – but in most cases the expectation is that the fiber project will generate enough revenue to repay the bonds. But fiber construction is expensive in big cities and the utility fee is needed to finance a network that will cost more than $1 billion in San Francisco.

The city’s rationale for considering this is to provide world-class broadband to everybody. This is a city that is in direct economic competition with cities in Japan, Taiwan and South Korea – and the city views fiber as a necessary component to long-term financial success. Comcast is the biggest ISP in the city and they have fast broadband today with speeds now up to a gigabit download. AT&T offers DSL plus has built fiber to large businesses and MDUs. Sonic has been building some fiber to residences in the Bay Area. And like in every large city there has been some fiber built by ISPs and CLECs to selected locations in the city.

But the city is concerned that a significant percentage of the public can’t afford fast broadband access today. The Blue Ribbon Panel notes that the government-sponsored fiber network in Singapore reduced broadband prices from $90 per month down to $30 – $40 today while speeds leaped to a symmetrical gigabit connection.

No NFL city has yet tried to build fiber and this proposal is going to meet a lot of resistance. Certainly Comcast, AT&T and other big ISPs will do everything possible to derail such an effort. The city says that they don’t want to directly compete with commercial ISPs, but if the fiber network really lowers gigabit prices to $30 – $40 that will clearly get most of the customers in the city.

I foresee all sorts of attempts to try to stop this project. The big ISPs are enjoying unprecedented support today in Congress and the FCC, and one ISP tactic might be to legislate against the project – either at the federal or state level. My fear is that a legislative approach might also stop more traditional municipal broadband projects. I would also expect to see numerous lawsuits from ISPs challenging the project. It’s such a new concept that it’s hard to envision the basis for such lawsuits, but I fully expect them. I can also envision a few citizen lawsuits trying to stop a mandatory new utility fee – picture forcing Comcast employees to pay to construct a competing network.

The final big hurdle will be in getting enough quality ISPs on the network to offer real customer choice. The few open access networks in this country have not attracted the many quality ISPs. The open access model works in Europe because the old state-monopoly telcos and cable companies have been forced into competing with each by the formation of the European Union. And perhaps quality ISPs will take a chance on a network in an NFL city. But in this country there seems to be agreement among cable companies to not compete with each other and it’s unlikely that we would see Charter, Mediacom or others stepping in to compete against Comcast.

This is a really interesting idea and it could be a viable way to get gigabit broadband to everybody in a big city. The city has not made the decision to take the leap forward, and if they do they will certainly face an uphill battle to make it work. But this could be the first trial in trying to bring the European open access model to the US.

When a Consultant Says ‘No’

Doug Dawson, 2017

One of my competitors recently held a webinar where they told a group of municipalities that they should never accept ‘no’ from a consultant who is evaluating fiber business plans. This is about the worst advice I think I have ever heard for many reasons. I think perhaps this consultant meant that one shouldn’t be afraid to be creative and to look at alternative ideas if your first ideas don’t pan out. But that’s not what they said.

Building and operating a fiber network is like any other new business venture and sometimes a new business venture is just not a good idea. This is why anybody launching a new business of any type does their homework and kicks the tires on their ideas to quantify the opportunity. A feasibility study means going through the process of gathering as many facts as possible in order to make an informed decision about a new opportunity.

The advice in this webinar was given to municipalities. Somebody giving this same advice to for-profit ISPs would be laughed out of the room. Established commercial ISPs all understand that they have natural limitations. They are limited in the amount of money they can borrow. They understand that there are natural limits on how far they can stretch existing staff without harming their business. They understand that if they expand into a new market and fail that they might jeopardize their existing company. My experience in building business plans for existing ISPs is that they are as skeptical of a good answer as a bad one and they dig and dig until they understand the nuances of a business plan before ever giving it any real consideration.

But municipalities build fiber networks for different reasons than for-profit ISPs. Existing ISPs want to make money. They also undertake expansion to gain economy of scale, because in the ISP world being larger generally means better margins. But cities have a whole other list of motivations for building fiber. They might want to solve the digital divide. They might want to lower prices in their market and foster competition. They might want to promote economic development by opening their communities to the opportunities created by good broadband.

These are all great goals, but I have rarely talked with a municipality that also doesn’t want a broadband business to at least break even. I say rarely, because there are small communities with zero broadband that are willing to spend tax dollars to subsidize getting broadband. But most communities only want a fiber business if the revenues from the venture will cover the cost of operations.

Sometimes a strong ‘no’ is the best and only answer to give to a client. Clients often come to me determined to make one specific business plan idea work. For example, many communities don’t just want a fiber network, but they want a fiber network operating under a specific business model like open access. That’s a business model where multiple ISPs use the network to compete for customers. Open access is an extremely hard business plan to make work. I’ve often had to show municipalities that this specific idea won’t work for them.

Or a commercial ISP might want to enter a new market and want to make it work without having to hire new employees. My advice to them might be that such an expectation is unrealistic and that over time they will have to hire the extra people.

My advice to clients is that they should be just as leery of a ‘yes’ answer as a ‘no’ answer. For example, every one of the big open access networks has an original business plan on the shelf that shows that they were going to make a lot of money – and those business plans were obviously flawed. If they had challenged some of the flawed assumptions in those business plans they probably would not have entered the business in the way they did. It’s a shame their original consultant didn’t say ‘no’.

I’ve always said that ‘dollars speak’ and any new business has to make financial sense before you can think about meeting other goals. Every business plan contains hundreds of assumptions and it’s always possible to ‘cook’ the assumptions to find a scenario that looks positive. I have created business plans many times for commercial and municipal clients where an honest look at the numbers just doesn’t add up. I’ve had a few clients ask me to create a more rosy forecast and I’ve always refused to do this.

I personally would be leery of a consultant that doesn’t think that ‘no’ can be the right answer for doing something as expensive as launching a fiber venture. Sometimes ‘no’ is the right answer, and if somebody tells you ‘no’ you ought to listen hard to them. It makes sense to kick the tires on all of the assumptions when you hear ‘no’ and to get a second opinion, if needed. But it’s important to kick the tires just as hard when you get ‘yes for an answer.

A New Model for Open Access?

Fiber CableThe traditional open access business model to serve residential customers has never worked in this country. I am familiar with the financial performance of most of the existing open access networks and from a purely financial perspective they are all failures. A few networks have failed outright like Provo. A few others have been able to generate enough revenues to cover annual operating costs, but most don’t even do that. And from what I’ve seen, none of the existing open access networks have ever been able to generate enough cash to pay anything towards the cost of building the fiber network, leaving the cities that build the network holding the financial bag for the initial investment.

There are a few reasons that this has never worked. First is that open access naturally drives ISPs towards cherry picking. Open access networks operate by charging fees to ISPs to use the network. If an ISP pays the typical $30 per month fee to use the network, they are not going to sell inexpensive broadband to anybody in the community. So when ISPs only sell high-priced products they don’t get enough customers and the city network owner doesn’t collect enough revenue to pay for the network. This has happened to every traditional open access network. None of them have signed up enough customers to pay for the networks and everyone who has built a network using this model ends up heavily subsidizing the open access network.

The other issue is that most cities have had trouble attracting very many quality ISPs. The whole concept of open access is to offer choices to customers. But most of the open access networks in the country only have a few ISPs, and even the ones they attract are often tiny, undercapitalized businesses. Attracting ISPs is so hard that there is one large open access network today that has been reduced over time to having only one residential ISP on the network. That’s not providing much customer choice.

But there are two cities looking at an alternative model. One is the small city of Ammon, Idaho, and the other is San Francisco. Both cities want residents to pay for the basic cost of the network. It’s an interesting idea.

In Ammon a household that wants broadband access will pay a tax levy of $10 to $15 per month and will also pay a utility fee of $16.50 per month. This means that each subscriber will pay $26 to $31 per month for the fiber network – a very similar charge to what is charged to ISPs on other open access networks. The Ammon commitment is voluntary and only those that sign up for broadband will pay the fees.

San Francisco is considering a similar proposal. There, residents would pay a monthly utility fee of $25 and businesses would pay as much as $115. In San Francisco this fee would be mandatory and everybody in the City would be assessed the fee. In an NFL city the fee probably has to be mandatory to assure that the network will be paid for.

Having customers pay a fee to the city takes the pressure off the cherry picking issue. By lowering what ISPs pay there is a lot better chance of having affordable products on the network. And that ought to result in more customers on the network.

But like any idea this one still leaves some open questions. For instance, how does the city make enough money over time to pay for the inevitable replacement of electronics or catastrophic events like storm damage? Or what does a city do if the ISPs don’t do a good job and customers don’t like them? The Ammon plan requires the payment of fees for a very long time, and small businesses like ISPs often don’t have the staying power to last for a long time. How will the business keep up with inflation – will the fees have to increase every year? And what happens if the city doesn’t get or keep enough customers to pay for this – will the fees go up for everybody else or will the city subsidize the network?

In a voluntary system like Ammon I also wonder what the consequences are for homes that change their mind over time. What if somebody has a financial problem and is unable to pay the fees? What happens when they want to sell their home – is this fee a tax lien of sorts? That’s what has happened to homes that buy solar power systems that are paid for over time. And what happens if a new buyer doesn’t want the fiber and doesn’t want to pay the fee? No doubt over time there will be legal issues to figure out.

The challenge to make this work in San Francisco seems much more difficult. It’s not hard to envision lawsuits from citizens who don’t want to pay the fees. And I can imagine a fierce battle with Comcast and the other current ISPs over the legality of a mandatory fiber utility fee. This seems like a concept that could take a decade of court time to resolve.

But the idea of having citizens somehow pay for the fiber network is an interesting one. Irrevocable customer pledges are a revenue stream that can be used to finance fiber construction. It’s hard to know if this concept will work until we see it in action. But it shows how serious cities are becoming to get good broadband. One has to think that if households are willing to sign long-term pledges to pay for fiber that it has to make a difference. I am sure communities all over the country will be watching to see if this works.

New Life for Open Access Networks?

CoH_City_seal_BlueGoogle Fiber and Huntsville Alabama just announced an interesting public private partnership. This is something that’s new for Google. In this partnership Huntsville is going to build and own the network and Google will lease connections on the network. Other ISPs will also be able to get on the network making this an open access network.

The details of the arrangement were not announced but there have already been a couple of interested parties that have made public records requests about the deal, so we ought to know more soon about how it will work.

There are a number of different ways to operate an open access network. For instance, a city can only own the fiber network and leave it up to ISPs to install the needed fiber drops and the customer electronics. At the other extreme a city could pay for everything. Since it’s been widely reported that Google uses some proprietary electronics my guess is that Google will be responsible for the electronics and the city for the rest. But we’ll have to wait a bit to see those details.

If Google does utilize a custom set of electronics it will be interesting to see how the city proposes to handle adding other ISPs to the networks. There are a lot of networks that would have a hard time handling different kinds of electronics mixed everywhere throughout the network.

The real question that everybody is going to want to know is if the city can make enough revenue from this arrangement to pay for the network. I’ve modeled open access networks many times and about the only way I can see for the network owner to break even with open access is if there are a lot of customers using the network.

And that is the biggest dilemma for owning an open access work. The big open access networks in Europe have a very high overall penetration rate because there are literally a dozen quality ISPs that compete on each network – basically multiple Googles. But if customer penetration rates fall below 50% it gets harder to see a path towards profitability for the network owner.

Fairly simple math can be used to demonstrate the dilemma for open access. If the network has a high penetration rate, say 70% or higher like happens in Europe, then the network owner can charge a relatively small fee per connection and can still break even. But should that same network have a small penetration, say 30% or 40%, the network owner would have to charge twice as much per connection to recover their costs.

The dilemma for network owners is that charging a high connection rate naturally leads the ISPs to cherry pick – that is, not sign up customers with low revenues that don’t create a good enough margin over and above the cost of the network connection. To give an example of this, if a network has a connection charge of $15 per customer, then some ISP in the market is probably going to be willing to use that connection to sell relatively low-price broadband, perhaps at $35 to $40 per month. But if the connection charge is instead $30 per customer, then no ISP is likely to chase those same $40 customer revenue opportunities and will only pursue customers willing to pay more.

This puts network owners in an economic bind. If they charge a low rate but don’t get a lot of customers they don’t make enough revenue to recover their costs. But if they raise the connection charge they force the ISPs to cherry pick and only sell more expensive products, and the network owner still might not sell enough connections to break even. The higher the connection charge, the fewer the potential connection that can be sold. It’s an interesting economic dynamic and one that puts all of the risk on the network owner.

I’m sure the deal is good for Google or they wouldn’t have signed it. It certainly relieves Google of a huge capital outlay. What others cities are going to be most interested in is if this a good deal for Huntsville. Most of the open access networks in the country have not done well for the network owner and it will be interesting to see if having a premiere tenant like Google will make a difference in the open access dynamic.

Can Open Access Work?

MLGW_Substation_Whitehaven_Memphis_TN_2013-01-06_006Today I am meeting with the Public Utility Districts (PUDs) in Washington State and they have been gracious enough to invite me to be the keynote speaker at their convention. These are the rural electric companies that serve much of the state outside of the major cities.

Whenever there is any listing of the fastest Internet speeds in the country the areas served by some of these PUDs show up among the fastest places because many of the PUDs have invested heavily in fiber. But they have a unique business plan because there is a legal restriction in the state that prohibits PUDs from being in the retail telecom business. This has forced them into operating open access networks where they build the fiber network and let other companies provide services.

No two of the PUDs have gone about this wholesale business in exactly the same way, and so together they provide multiple experiments on ways to operate a wholesale open access network. I know several of the PUDs well and they have one universal problem – no large, well-financed service provider has agreed to offer service on their networks. No big cable companies or telcos or anybody you ever heard of wants to serve the many customers on these fiber networks. There are a handful of connections sold to companies that serve large businesses, like Zayo and Sprint, but no bit company that wants to serve smaller customers.

What is lacking is vigorous competition on their networks from multiple companies willing to serve residents and small businesses. And that is what open access is supposed to bring. Instead, most of the retail service on these networks is provided by local ISPs who took advantage of the opportunity to reach more customers. In many cases the local ISPs were so small and undercapitalized that the PUDs had to assist them to expand onto their networks.

There are not many other open access networks in the US. One of the largest ones was in Provo, which struggled with the model and eventually sold their network to Google. I was privy to Provo’s books and they could not find a business plan model that would make their business model cash flow. But if we look outside the US there is another great example of how open access can work if done right. Europe has a number of cities that have built fiber networks and invited ISPs and others to serve customers. In Europe this has been a big success because numerous service provider show up to provide service. Some of these providers were the former state monopoly companies that were unleashed to compete after the formation of the European Union. But there are also new competitors there akin to our CLECs and ISPs.

The big difference between US and Europe is that here none of the incumbent competitors are willing to operate on somebody else’s network. I can’t think of one example in the US of large cable companies competing against another one. And there is very little competition between the big telcos other than some fierce competition for some giant government and business accounts. Here in the US the PUDs have only been able to attract small local ISPs to operate on their networks. For the most part these ISPs do a good job, but they are small and have the problems that all small telecom companies have.

Many of the PUDs are in the uncomfortable position of only have one real service provider on their network. Should the owner of that business die or just go out of business a PUD could see most of their network go dark and all of the residents and businesses in their towns lose their fast Internet.

Anybody who understands telco finances instantly understands why this model is so hard to make work. A company must spend a lot of money to build a fiber network and then can charge only relatively small fees to others that use it. A typical revenue for wholesale access to a fiber network is in the range of $30 per customer per month and that is really not enough revenue to pay for building and operating the fiber network. By comparison, most triple play providers have an average revenue per customer north of $130 per customer per month.

The PUDs built the fiber because they are in rural areas where nobody else was going to do it. Their communities have already benefitted tremendously from the fiber. But they have their work cut out to keep this going, and I am sure they will figure out a way to do so.