Apparently CNET News took some interest to my blog posting on the salesforce.com outage Monday. Looking back at my article that's getting this exposure, I'd like to clarify a few things. Salesforce.com has been a big help to our business. Prior to salesforce we used a home grown system that was a nightmare to keep up to date and progress. No other CRM system that I looked at had the features and extensive API to create custom integrations and services. They are also constantly releasing new feature updates; about 3 a year. We have been able to integrate and synchronize our local ecommerce and account system with salesforce via their SOAP API which is hands down better than the Microsoft CRM API (at least with version 1.1 which I last looked at). They also have designated "customer success managers" that I can email or call to comment (or complain). My manager has always been very helpful and responsive. Using their support system has taken a hit lately though. We are an enterprise customer, but with no special support plan. What used to take a day or two to reply to an issue now can take up to 2 weeks. However a call or email to my success manager would likely rectify that.
There also seems to be some contradicting information regarding salesforce's $50 million mirrorforce data center. I just got off the phone with my success manager today, and he said that they moved to a larger data center in November, and are building a duplicate data center on the east coast (hence the name mirror force). It wont be until the coming months that this redundant data center will be brought on-line. I was under the impression that the move to the new data center this past November was going to improve reliability and performance. We've been a customer since August 2004 and both performance and reliability is as bad as it ever has been after this move contradicting my expectations. I'll be giving salesforce another chance to make true their promise on increased reliability and performance once the redundancy of mirrorforce is in full effect. I will be speaking to one of the salesforce.com architects this Friday about their future plans and will update you on that discussion here.
Let me also clarify why up-time is so important to us and why this latest outage was such an inconvenience. We rely on the API to carry out business transactions that originate from sales in our ecommerce system, and creation of new accounts and contacts when a customer downloads our product to evaluate. We rely on the user interface for incoming support and sales calls. When a customer calls we look up their information in salesforce. We also rely on the user interface to respond to our customer's support request, which for customers who have paid additional for support and maintenance is typically within 2 business hours. With just the UI, we can respond to cases, however the comments don't make it into our local database unless the API is working. Therefore on Monday, customers who opened a case in the morning didn't get a response until the next day. We have been forced to designed our system to accommodate outages and slow performance as best we can without developing a CRM system of our own. We have a local database that mirrors some of the salesforce data so that when customers access information from their portal, they are actually viewing a local copy. Luckily this was deployed at the beginning of December which was prompted after their awful week of outages after the November data center move. This really saved us Monday. Without it, our customers wouldn't have even been able to submit or comment on a case, and would have just seen an error message. Sales transactions and new leads would have never made it into the system.
With all these problems, what really dissapoints me is what the salesforce.com's management has to say about this outage. The email message I received yesterday claimed no more than 30 minutes of intermittent access (that's not true) and CEO Benioff himself is quoted as saying it was a minor issue. EWeek's article about the incident quotes Bruce Francis, president of coorporate communications as saying these reports of outages lasting more than an hour are inaccurate. Hah!! We couldn't get in for well more than 2 hours and it was slow as hell prior to the outage. Don't believe me? Read the comments on salesforcewatch.com's recent post. Perhaps managent was misinformed. Then there's the 7 hour API downtime. Why all this downtime? Here is the explanation:
When the system became unstable, we disabled API access as part of our efforts to restore performance. After the database was restarted and the system was stabilized and tested, we were able to restore API access.
While the system has a whole was only unavailable for about a half an hour, API access was generally less available as we attempted to restore full system performance
We apologize for the inconvenience.
I also questioned the future of On Demand applications. Our system is really at the mercy of 3 providers. Our web host, maximumasp who hosts our web server, salesforce, and our own ISP. If our ISP is down (which it rarely ever is) we can go home and still get work done. Our customers are not affected at all. Our website is almost never down and our host claims nearly 100% up-time. Never in the 4 years we have been a customer have they been down for over an hour. Salesforce is down or so slow it's unusable more often than I'd like to think; at least once a week. Some of my employees that use the system much more than I do would even go further and say that it's unusable a few hours a week. If only we could install salesforce on a server hosted on maximumasp, our problems would be resolved. But that goes completely against their business model! With that said, I think if Microsoft CRM can catch up with salesforce.com in features, extensibility, and ease of use, salesforce better watch out.
