November 7, 2007

Here's the really hilarious part - Navisite positions itself as an expert in Disaster recovery -

http://www.navisite.com/sublevel.aspx?id=1887

as they point out: In today's world, Disaster Recovery (DR) capabilities are important for virtually every organization, because those that cannot recover quickly often do not survive.

actually I'm kind of surprised they haven't edited that to say - "any business can survive downtime of only a few days, but when you are offline for a month, that is a DISASTER..."

I have to agree with TG 100%. The Surebridge migration was quite a long haul of working lots of weekends and nights. And there have been others in Vienna, VA and Atlanta, GA as well. I can tell you all the engineers involved in those moves worked 20+ hour days and gave it there all. Unless things have drastically changed in the past few years, engineers working on the problem 24/7 until it was fixed.

Can you tell us which plan you have or had from linuxwebhost.com? Is it the $9.74 plan, $14.99 plan, $29.24 plan, or the $44.99 plan, for your business critical web presence? And whats the service level agreement for downtime?

I have 2 questions for Cynthia Brumfield.

1. Have you backed up your files yet?
2. Did you change providers yet?

It contiunes at Navisite. After going down on Sunday the 28th for 7 hours for a move that didn't happen then going down on Saturday night the 3rd and didn't come back up until 2am on Friday the 9th.

We spent Friday morning backing up all data so our files would be fresh. Then tried to start to address emails and damage control. We started all our marketing to try to start to bring in revenue again. Then put in calls to get quotes from other companies on dedicated servers. Most can't get us information until Monday.

We are now down again. Saturday the 10th we went down for the third time at 1 in the afternoon. It is now 11pm and still nothing but a email reply saying they are looking into it.

At this point we are done. Our company has lost to much and will put what resources we have left to get on with another service before we are out of business completely.

Peter,

Well, apparently the smarter people at Navisite overruled the attorneys and had CEO Arthur Becker issue an apology late today. About six days too late, which is one of the dumbest PR moves in corporate history. And what an apology it is.

See http://www.navisite.com/sublevel.aspx?id=2017

Becker basically apologizes for the "inconvenience and disruption" but notes that on the bright side of things, the new servers are good for the environment. I almost choked on my Diet Coke when I read that.

One fellow (http://5dollarbackup.com/blog/), who started out so optimistic earlier this week, said the "apology" made him want to gag.

"We're supposed to be happy because the new VIRTUAL servers they replaced our dedicated machines with are better for the environment? Who cares about that when their business is destroyed? This was not an 'inconvenience' or 'disruption', it was a death blow to hundreds of small businesses."

"This is another outrage to add to the pile," he said.

I agree. Navisite's denial of reality and seeming indifference to the pain -- or as some Dilbert-like Navisite executive noted earlier today, the "pain points "-- it was inflicting (not to mention false public statements) is what fueled most of my outrage. It still does, and based on this guy's similar reaction, nobody affected by this crisis is buying the company's sociopathic spin.

Some investors, however, ARE buying it. The company's stock closed up today, a day when most companies on the Nasdaq were brutalized.

But, this historical outage is merely a symptom of something terribly wrong at Navisite and sooner or later investors will figure that out. I've learned from past experience that giant red flags such as this, whether we're talking about people or corporations, are merely the precursor to more symptoms of pathology.

Whatever Arthur Becker has done to keep the company's stock afloat, he won't be able to keep it up. Not when news of the lawsuits hit. Not when Navisite issues its next quarterly earnings and the numbers are bad. Not when Navisite's rivals move in for the kill.

As of yesterday, all three of my sites were back up, though not consistently, and about 5 emails came through. I quickly pulled some stuff off my sites that I hadn't backed up, and went to Yahoo for their Small Business hosting services.

My original hosting service (leased from Alabanza, I guess) was LinuxWebHost. They have been unreachable by both phone and email since the move began, and NaviSite has been almost as unresponsive (though I occasionally got a live person who just offered generalities). What I couldn't do through either of these means was cancel my hosting services. I've instead instructed my credit card company to stop any future charges to my account and to do a chargeback on my last hosting fee.

I encourage anyone else caught up in this debacle to take this same recourse.

I don't use Navisite so have the luxury of being able to comment objectively. Concerning the invisibility of management, and their related unwillingness to apologize, I assume their lawyers are advising them that every single word they speak or write in public can be used against them in loss-of-business lawsuits.

Navisite is not going to skate on this one and no matter what kind of service and technology they provided in the past, that's history.

Despite the company's public statements that it has beefed up customer service support and has resolved all issues, that's simply not true. In other words, Navisite is lying, as some of the comments posted here attest.

I've also continued to receive emails from companies whose sites are still not up or whose sites are up-and-down. Here's one email I received about 30 minutes ago, or around 10:45 a.m. on Friday. I received permission from the sender, an ecommerce provider, to publish his remarks:

"Once again all of my company's approximately 40 websites are down again this morning. It is now almost official - that I have lost an entire week of new sales for my company on what is traditionally one of our biggest weeks of the year. I am going to take the advice that some people are suggesting and contact the law firm of Wolf Popper in NYC about initiating a class action suit against Navisite. They have destroyed the income and reputation of thousands of small business people."

One thing this person flagged for me in an earlier email, a problem that I haven't read about so far, is that Navisite promised that it was caching everybody's emails so that when the sites did come back up, historical emails would become available.

Um, no. Few emails were cached. When my email did come back up, there were a weird handful of emails cached, including the first one, to my developer, that I sent last Sunday entitled "Help -- My email and web sites are still down." Pretty amusing

With respect to the Alabanza migration itself I can't comment. I wasn't there to know what types of decisions were made and in what context, but it's atypical for their history of change control. The Surebridge data center migration into Andover in 2004 (as much as some engineers may have hated that summer) was a well-planned and well-executed move over the course of 12 weeks. Clearly Alabanza was more rushed. I suspect the thought of paying another month's rent in Baltimore was a key factor in pulling the (fateful) trigger. Navisite has been guilty of that in the past. As far as competence, however I have to come to their defense. Over the course of 7 years I watched a flury of companies and engineers ebb and flow as Navisite came into being (primarily from 2004 to now). Some of the technology that was deployed was the best I've seen, and those that managed that technology were equally impressive. What makes this move embarrassing is the 'perfect storm' effect. If there were 5-10 servers that needed triage, like a 2-car accident, those affected would have gotten the needed attention (i.e an insurance check) and all parties would be as happy as could be with the results...but picture a 200 car pile up, where everyone has the same insurance carrier. There's no way the insurance company could cut everyone a check at the same time. Navisite has been successful technilogically by offering the best value for the dollar that they can. The problem is that their success (financially) stems from playing the odds. They have a large contingent of off-site engineers capable of troubleshooting and mitigating issues from a remote location....which is 98% of the troubleshooting equasion, but when you need hands on (because the hardware is down or inaccessible) the limited, in-house head count becomes apparent...as it has in this case.

When the dust settles and all systems are running, those that choose to stay WILL be satisfied with the service, support and performance that Navisite offers.

I am a former employee...and as much as I'd like to chuckle at the misfortune of a company I was happy to leave I do have to speak frankly about unjust comments from those affected.

There was undoubtedly a management decision made that (if executed) would have made that person, and Navisite a hero. Unfortunately a series of unforseen events caused a snowball effect and now the possible hero is a scapegoat.

Navi has come a long way, even since i left, so I'd hate to see them crash NOW. For those of you that survive this, have an open mind for the future. You will be satisfied with what you ultimately end up with...and I can guarantee you this won't ever happen again.

My website is up but not secure, which means i'm still losing business. They say the SSL certificate is on the site, but it doesn't redirect to https. We have been emailing for 2 days now and they just keep telling us to put in a support ticket, which was already done.

My whole reason for posting is to alert anyone that has an ecommerce site to check your security.

Also, how do I sue these people? Any suggestions, anyone else on board????

One of our servers went down again. its 6 hours and I tried everything, emergency@ rebootrequest@ nocmonitor@, filled the form, called all support number on the site, yet, nothing..

There is only one guy that I give him credit for trying to help and he is Pete Castello.

I saw that Hiawatha Bray had pretty much given Mr. Sinha and Navisite a free pass when he used their estimate in the headline, and said that Alabanza claimed 165,000 customers when in fact it was a Navisite press release, no doubt supervised by Mr. Sinha himself that made that claim.

I reprint my letter here in case the Globe chooses not to print it.

---

Hello Hiawatha,

I have followed your columns over the last many years and have been generally very impressed.

I must say though, that you gave the Head Flack from Navisite WAY too much benefit of the doubt (you were way too nice) when he baldly and provably lied to your face about how many sites were affected.

It was not Alabanza who claimed to have 165,000 customers (web sites), it was NAVISITE itself, in an August 13 press release.

See http://www.TotallyFixed.com to see said press release.

You hinted that he might be full of it, but it was an easily provable lie, and part of the pattern of lies and deception and sheer incompetence (bordering on criminal negligence) that Navisite has displayed throughout this entire fiasco.

Just thought you ought to know.

I had over a hundred sites down myself, for over 60 hours.

This is somewhere between an emergency and a national disaster.

165,000 companies are a lot of companies to be out of business simultaneously.

All this could have been prevented too, but that would have been "too costly", like hiring customer service reps from Massachusetts.

It was nice talking to New Delhi whilst sitting in Massachusetts myself, trying to reach a Massachusetts company.

This needs to be pressed further. Google the subject "Navisite Alabanza" and you'll quickly get the full scope of the crisis.

There are sites/companies that are STILL not up and running, and it's Thursday. They've been down since Saturday morning at 1am.

Just thought you should know. The Globe was slow to respond to this story (I called the business desk Monday, saw your article today, that's not exactly timely) but I am optimistic that you'll finish strong and give these lame-o's the investigative beating they deserve.

Thank you for your time and consideration. As an affected party, you can call me for my input if you wish.

Be well,

David Caputo
President
Positronic Design
Holyoke, MA
(Alabanza customer since 1999)

Wow this has been just a crazy situation and nothing less then a total disaster for many companies not only ours.

We have a (or had) a dedicated server with Alabanza that we pay a premium price for. Our situation is a bit different as all the sites (50) on our server are all our companies.

We did get the email and notice the first time that the servers were to moved to the Navisite location. Then we recieved an email that the date was changing and they would send us notice. The next few days our server ran very slow and had many problems with emails. Then our site completely went down for 6 hours. When we contacted Navisite/Alabanza support they said it had been shut down for the move that didn't take place. They would restore the server. At that time they couldn't give us any details on the new date.

On Saturday night I was to find out that our server was down and the move was taking place. We were waiting to get notice, so we could pull the most current information from our SQL db's. Needless to say this didn't happen.

It is now Thursday the 8th at 3:45pm and we are still 100% down. We also spent countless hours on the phones and sending emails. All of which went un-answered.

As stated we operate about 50 sites and a couple large e-commerce sites such as netshades.com We are losing thousands of dollars every day in lost revenue from that site alone.

We are seeing the updates that over 90% of the servers are operating and we are very happy for the companies that are up as we know what they have been through. We just keep wondering when will it be us. We continue to sit on hold and get canned replies.

What happened to the statements made by Navisite that this move would take us down for at the most 10 hours? It has been nothing less then lies and mis-information from the begining.

All we can do at this point is contiune to hope that our server will be back online soon as we have employees and overhead and 0 dollars coming in.

I think that the only real solution is to pull our sites out of the Navisite infrastructure. Also, upper management needs to clean house - starting with Mark Clayman. There is plenty of responsible competition out there that would pay more attention to their clients than Navisite has.

Our small publishing company has not seen our Web site since Sat. November 3. Today is Nov. 8. It is gone. Toast. Neither have we received a single e-mail since. Having listened to the many conference calls that were held earlier in the week, which have now stopped, I wrote down some of the urls of companies that Navisite has put in the grave. I have checked them repeatedly since, and they are all still dead. It is absolutely pathetic. The nightmare gets worse and continues for thousands of us. The bastards. There are many worse words to describe Navisite and what they have done to our livelihoods.

PS... I think that NaviSite's statement that they have 60% of sites back online is a bunch of crap.

My NaviSite-hosted website is still down, and NaviSite still won't return my phone calls or messages. I alerted CNET this morning about the ongoing outages and also spoke with Elinor. Cynthia, thank you for continuing to document this debacle.