Healthcare.gov is the website where you can review and purchase health insurance under the Affordable Care Act (ACA, also known as “Obamacare”). Based on what state you’re in, you may be directed to a state-run website (www.CoveredCA.com for Californians) or provided information directly (in cases where a state-run site isn’t available, typically because the state government has issues with ACA provisions). It launched on October 1, 2013, and immediately ran into problems. After a couple of weeks of daily embarrassment, the Obama administration announced a “surge” to fix the problems by November 30. Although the site was still far from “fixed” by that date (serving only about 80 percent of requests successfully), the administration put on a brave face and announced it had met its goal. How did this very public failure happen? More important, what can you learn from it?
The ACA was signed into law on March 23, 2010, giving the government a little more than 2.5 years to build the healthcare.gov website. A lot was made of the fact that, during the election campaigns of 2008, the Obama campaign proved to be more technologically capable than its opponents, something many believe to have been a key element in Obama’s win. Unfortunately, development of the technology used in the 2008 and 2012 presidential campaigns wasn’t subject to the convoluted mess of regulations that dictate how the government procures goods and services.
What happens is you have a relatively small number of companies that have the expertise and resources (such as lawyers) to deal with government procurement. They’re the ones who write the proposals and are awarded the contracts and, in turn, subcontract with the companies (mostly large companies like IBM and Oracle, and some upstarts, like MarkLogic) to do the actual work. If you look at the publicly available information, you’ll see there were 50 direct contractors providing services related to the ACA implementation, with payments totaling about $450 million.
First lesson: The more entities involved in producing something, the greater the likelihood of failure. This seems like common sense, and yet the governmental procurement process virtually guarantees multiple companies (and layers within each of those companies) will be involved, making the kind of clear, rapid communication at the heart of nimble software development a virtual impossibility.
I’m amused when the government points to successes like the health care information systems that power Kaiser’s health system, or consumer-friendly websites like Amazon.com, and seek to emulate them. It took Kaiser 10 years and more than $4 billion to develop its system. Amazon has been refining its site since 1995. The 2.5-year schedule for healthcare.gov was ambitious to begin with. Even more so when you consider that both Kaiser and Amazon have complete control over their internal systems, and a lot of leverage when dealing with outside interfaces (for example, getting test data and images from outside labs).
On the other hand, while the public interface to the health insurance marketplace was being built from scratch, there were a large number of legacy systems with which its builders had to contend (Medicare, the Medicaid services in each state, insurance companies and so on). From experience, I can tell you that dealing with a legacy software system has two common problems. First, the interface may be primitive or nonexistent. These legacy systems were never designed to share data, and particularly not in real time. A lot of them are batch-oriented, which means information is usually between zero and 24 hours old, since updates (batches) are run overnight. Second, interfacing with outside parties can reveal hidden problems due to unanticipated and unfamiliar requests of the system.
The take-away: Modern websites aren’t as simple as they seem, and getting data in and out of another system isn’t as easy as you might think—and it gets harder the older that system is. In the programming biz, it’s known as a SMOP: a “simple matter of programming.” It’s a sarcastic way of saying that what looks easy and works well takes a lot of hard (and thoughtful) work behind the scenes. It’s true that it’s easier than ever to build a great website, particularly due to the availability of cheap cloud infrastructure (such as Amazon Web Services), which used to cost millions to acquire and operate in the late 1990s.
Not long after healthcare.gov launched, a small group of programmers made news when they claimed to have built a website that did everything healthcare.gov did in a couple of weeks. First, that ignores the very real problem of legacy system interfacing. But second, it ignores the problem of scale. It’s relatively easy to create a website that will handle one or two simultaneous visitors (although even going from one to two has its hidden pitfalls if you don’t understand concurrency). It’s much harder to build a site that can, on its first day of operation, handle thousands—or more—of simultaneous connections. Most sites do a “soft launch” so they have time to see and correct problems with loading as traffic increases over time. Not healthcare.gov—having a fixed and very public launch date virtually guaranteed a disaster.
There were certainly other idiocies at work as well. A project of this scale takes months of testing, and testing is the first thing to get squeezed or omitted. There was no experienced “integrator,” the group responsible for making sure all the individual pieces work together. There was an expectation that everything would work right out of the gate. And there was a lot of denial as the project got closer to deadline. It was an embarrassment for a President whose team seemed to be so good at technical stuff.
Eventually,healthcare.gov will work as well as any other high-traffic website (or nearly so). But its initial technical problems created a huge political nightmare for the Obama administration. Lest you think these problems are confined to government, Standish Group International found that, in 2012, only 10 percent of IT projects with a value of more than $10 million were successfully completed on time and within budget.
It’s 2014 (or will be shortly). Drop me a line at email@example.com and let me know what topics you’d like to see covered in this column during the coming year. It’s more fun for both of us that way. Happy New Year!
Measles, mumps, chicken pox and other childhood diseases of yesteryear are making headlines again in the United States. Formally prevalent viruses and bacterium affected thousands of lives before a sc...
Located at 1410 Neotomas Ave. in Santa Rosa,NorthBay biz magazine is a monthly business-to-business publication covering Napa, Sonoma and Marin counties. This year, the magazine is celebrating 43 years of continuous operation. It originally hit the stands in 1975, when it was called Sonoma Business, and only covered Sonoma County. Norm and Joni Rosinski and John Dennis, acquired it in 2000 and changed its name to cover an expanded market. Today, the magazine is part of Amaturo Sonoma Media Group. More here..