I have a client who has been buying a cloud service for about two years and has a number of reasons to be unhappy with the service. I’m not going to name the specific service because there aren’t many vendors in this particular space. But the issues my client is experiencing look to be common with a lot of cloud services.
My client buys this service to resell along with other services to his own customers. His number one complaint is that he never knows what service he is going to wake up to each day. The cloud software will be working great one day and then the vendor will implement a software change and all of a sudden he gets calls from his end-user customers that things have gone wrong. And inevitably the problem turns out to be something that the cloud service vendor has introduced into the service as a supposed upgrade or improvement.
This is not a new phenomenon and anybody that purchased a voice switch, a cable TV headend, or a complex billing system can remember back to the day when this same sort of thing happened all of the time. Carriers would shudder each time that they got a software update from a vendor because it often caused more harm than it did good.
And the industry learned how to deal with this problem. First, carriers started to insist that vendors build test labs and that they try out new software first in the lab rather than foisting it onto the lab of end users. Second, they insisted that software vendors update software in discrete releases so that each carrier could decide if they needed to install a new update or not.
I remember a time in the late 90s when CCG routinely recommended that our clients not install software updates. There were so many problems with new software releases that we found that it was better to let the update hit the world and to let other carriers debug each new software release. My clients would purposefully fall numerous software upgrades behind, but as long as they weren’t experiencing end-user issues they were happy.
But now my clients are starting to buy services in the cloud, and in doing so we have gone back to the 90s all over again. The biggest problem with most cloud vendors is that they only run one version of their software – the latest. The vendor will update the cloud software and every one of their customers will have the same version. This certainly makes life easier on the cloud vendor.
But unless the vendor has amazing software coders that never make mistakes (and that is never going to happen) then the vendor can release an update that has dreadful bugs in it and the test lab for these bugs become the end-user customers of the carriers. A carrier might not even realize there was a software update until they get complaints from their own customers. But now the situation is much worse than the old days, because one of the most common ways to fix this sort of problem used to be to reinstall an earlier version of the software that they knew worked right.
I guess that cloud service providers need to learn the same lessons that the other vendors in the carrier industry learned a few decades ago. Just because software is in the cloud doesn’t change good software practices – in fact it makes them even more important. A software vendor that uses end-users as his software testing lab is going to get a horrible reputation and in the long run is not going to keep customers in the carrier space.
And so I hope that software vendors would implement the same kinds of changes that the industry forced vendors to implement decades ago:
- It should be the carrier’s choice about accepting a software upgrade. Updates should never be automatic. This means that the cloud vendor needs to keep multiple versions of their software available online.
- Software vendors need to maintain a test lab of some kind. Most software ultimately controls hardware and the vendor ought to have a lab on which to make certain that changes in the software do what they are intended to do while not screwing up something else.
A friend of mine is a music teacher for a private school, wearing a number of hats. For one, she is one of the resident authorities on music… Secondly, with her degree in Telecom, she occasionally serves as an authority on telecom issues when they arise.
Similar anecdote — Someone had the idea to change from housing databases on-site to housing them off-campus with a telecom vendor. On the one hand, it meant that the vendor could worry about upgrades and maintenance, a thankless job requirement that no one wanted to do, and a lot of other tasks were removed from the job of folks that are essentially non-computer people.
The problems came on a number of points… (a) the vendor did not care about whether downtime and upgrades were convenient to the client (i.e., end-of-quarter grade inputs), the vendor only cared if the dates selected were convenient to the vendor — and with a number of bigger, more crucial clients, I doubt whether the school was on anyone’s “priority” list.
When the connections went down, coincidentally at the end of a marking period, repairs by the vendor were not a big priority for them — their priorities were to the Federal Government, hospitals, and other large-scale clients, etc., not to the little private school. Still, the vendor’s protracted outage caused the school all sorts of heartache.
Bottom line, after six months of this stuff, their databases returned home.