If anybody has recommendations for a new hosting provider that I can direct people to, I'm all ears. As it turns out, we are in the midst of a redesign (much delayed by me) of all our sites and we're doing some sort of new, flexible, Web 2.0'ish Ruby-on-Rails thing that might not be helpful to most people looking to flee Navisite. Post a comment with recommendations.
Just as I planned to organize a convoy to Navisite's corporate headquarters, our sites came back up! I understand from my developer who is on a 4:00 conference call with the company that many sites are still out and Navisite is still coping with routing issues.
But we're downloading the files as fast as we can in the event the sites go down again. That way we can quickly move to another hosting provider if need be.
Ironically, I began to get phone calls from various reporters this afternoon, who were waking up to what Navisite has done. So, expect a spate of articles post-fact.
There are a lot of lessons to be learned in all of this. First, always keep back-ups of everything away from your web hosting provider. Secondly, losing access to email, client projects, web sites and other Internet-related tools causes a form of insanity.
At no point since Sunday have I felt "OK." I always had this little crazy feeling humming around inside, wondering who was trying to reach me, whether clients would be upset that they didn't receive the stuff I promised, whether I was losing too much ground in my forward-looking efforts (which are already hideously behind schedule anyway.)
Third, I appreciate the vast amount of small businesses that depend on the Internet. Most of the Alabanza customers purchased by Navisite are small businesses or web hosting resellers who service small businesses. A lot of people have felt a lot of pain over the past four days and I've received dozens of emails from folks who lost revenue.
One person estimated that her business lost $20,000 in sales over the past four days, which is a lot of money to a small business owner. More importantly, some of Navisite's customers have lost customers themselves, which is a kind of permanent loss.
Finally, and foremost, there's a big lesson in all of this for Navisite, which, I suspect, is about to go through a world of hurt. The immediate pain will be the rapid loss of customers burned by the migration. There will also be class action lawsuits, no doubt. But the company has tarred its reputation to the point that it too will experience a permanent kind of loss -- opportunity loss, in the form of big corporate customers who will steer clear of the company, and reputational loss, which is extremely hard to ever recoup.
My trip to Boston is now off. If there are others out there with web sites that are still down, and you're in the Boston area or plan to visit Andover, I'd be glad to provide a platform for your retrieval of servers and files from Navisite.
To Navisite customers whose sites and email are still down, I feel your pain.
To my regular readers, again, I apologize for this weird interlude. With no real ability to work on my regular business, my only outlet was this blog. I'll get back to the regular programming schedule tomorrow.
Update: Several folks have commented here, via email and other sites that we got what we deserved for going with a cheap provider and not keeping back-ups separate from the hosting company. Well, Alabanza wasn't a cheap provider -- they were actually on the high end of the scale. One reason they could charge more is because one of their selling points was strong redundancy. They stressed the value of their back-up system, which gave customers comfort in knowing that redundant files were in place in case of a hacker attack. No one, however, guessed that Alabanza/Navisite itself would be the cause of the problem, so these criticisms are slightly off the mark. Be that as it may, as Navisite as shown, you can't trust your hosting provider to create redundancy in the event of a crisis.
Cynthia Brumfield at 4:40 PM|Comments(30)
Perhaps the company should have only done a few servers at a time instead of doing them all at a time. At least those who were down would have been few and it would have looked more like a well contained situation instead of the nightmare it turned into. Wow. I have a feeling that their business is going to suffer greatly for this mistake. I'm glad I don't work for them because I may be out of job due to (future) lack of profits.
Posted by: Jim in Arizona at November 15, 2007 11:28 AM
Navisite repOrts between 85-90% of the sites are back up. What they are not syaying is those sites are up at a new server company after they left NAVISITE.
My five sites are up - ELSEWHERE!
Posted by: Mark Amtower at November 12, 2007 3:08 PM
Uh Mike. Please note that your comment is directed at the wrong person. The person with the comment you've mentioned goes by the handle Jean. Poster information is at the bottom of the comment. Thank you.
Cynthia, could you please correct Mike's reference in his comment. Thank you.
Posted by: vicory at November 11, 2007 6:09 PM
Posted by: vicory at November 7, 2007 5:06 PM [ed.note: Mike misunderstood who posted the comment he cites. It was not vicory but "Jean."]
All that you have said and done over this situation should cause YOUR clients to DROP YOU as it is obvious to anyone with a little knowledge of websites and how they work, that you do not have the FIRST CLUE about the business that you are in. The total incompetence that you showed your clients by not having your own backups or even understanding the need for your own independent backups, by not having a contingency plan for server interruptions, by acting like a child who's daddy has abandoned her during this incident, in short by not doing anything you should have done and everything you should not have done, you have proven to everyone that you are incapable of handling even this minor situation, and this was and is a MINOR SITUATION as pertaining to Web based businesses. Grow up and take responsibility for your own mistakes instead of wasting your time ranting and raving insensibly like the immature brat that you seem to be. And get out of any web based affairs until you at least gain a basic understanding of what is needed and how it works.
Reply:
Sir, in my opinion, you are a dickhead! 165000 websites down is NOT a minor sitution to NORMAL thinking people. You must be one of those "I am more holy than thou" types, or do you work for the Bush administration? In either case, you would make a great politician. In addition, I would venture to say that you are one of those buttheads that never had aything like that happen to you, but you are more than willing to judge eveyone else from a utopian world distance. It is high time that YOU grow up and smell the coffee!
Posted by: Mike at November 10, 2007 2:25 PM
WOW... What a nightmare. Right in the middle of our holiday season they decide to change horses in mid-stream, but forgot the second horse wouldn't arrive for a few days. No back-up servers to cover the downtime was a big surprise to us all. Let's hope they have learned a lesson. We have all lost income during this untimely goof up.
Posted by: Charles Dittebrand at November 9, 2007 2:25 PM
20k in rev and 5k in income in two days???
According to your numbers you are running a 300k a month (3.6M a year) in rev and 75K a month (900k a year) in income business on $400 a month of servers?
You and Navisite are in stiff competition for the "Incompetence Award".
Posted by: Shocked at November 9, 2007 1:41 PM
Our company has had email since Tuesday, but our website was down. The website came back up yesterday, but then, as of sometime last night prior to 10 PM, we lost email. I guess you get one or the other with these clowns. I'm assuming NaviSite is run by former Bush administration officials. They'll probably get some kind of medal when this is all over with (if it ever is).
Posted by: Michael Ryle at November 9, 2007 10:20 AM
If any of you are wondering why none of the main stream media has picked up this story I can tell you. I contacted the Associated Press and they do not consider this to be a story worthy of national interest. If you disagree as I do I recommend you start calling the closest AP desk to you. You can find it at the AP's website AP.org, then click on Contact Us, they on the right hand side select your state and you will get a listing of all the bureaus in your state. Call them up and ask for the Tech Writer. I spoke directly to the tech writer in Boston and he was the one who told me the editors did not consider this important enough to cover nationally.
Posted by: Robert K at November 8, 2007 5:25 PM
Jay Keating (jkeating@navisite.com) is the "Project Manager" for this enture fiasco!
Posted by: Nate at November 8, 2007 3:41 PM
Navisite's latest update (at 7pm EST on 11/7/07) finally admits the truth of their incompetence:
"We have now mitigated all outstanding issues that were holding up progress and are steadily bringing up more servers live. More than 60% of the servers are now on line, and we currently estimate that all servers will be live within the next 24 hours."
In other words, after 110 hours (!), only 60% of the servers are online.
Posted by: francis at November 7, 2007 9:16 PM
Navisite has caused major problems for many independent and small businesses.
Jean, there is REAL money involved. A third party, who is in a trusted and paid position, that threatens my business cash flow is going to get the pointed end of the stick. I'm glad to hear this is a MINOR SITUATION for you Jean. There are plenty that will say differently, though. Have you ever considered what might happen to the economy if there was a massive DNS outage? That is, no basic Internet as we know it. Now shrink this effect down to the local, more personal economy. Then consider, what happens to your companies pocketbook during those days of lost cash flow? How much does hosting with such a provider truly cost? Is it just the number on the monthly invoice?
Jean, there are REAL customers on the line. A third party, who is in a trusted and paid position, that threatens my business credibility is going to get the pointed end of the stick. People who depend on web hosting take downtime very seriously. It is a direct reflection on the professionalism of the services that they provide. Now, insert a third party whom directly impacts your professional image. It doesn't take long for you to a) wither due to third parties killing your image, or b) you contract a different hosting service , or c) you manage the hosting yourself.
Jean, for someone who seems knowledgeable about web hosting, you should be well aware that bringing up site backups through a secondary provider is not necessarily a trivial task. Having a "hot" backup host available is also not necessarily an option to an independent small business due to the incremental costs. These things must be planned for with the execution taking valuable time away from the core business activities. Having to be forced into such contingencies would make any business owner upset. Navisite is paid to handle such hosting details as a professional services organization.
And Jean, Navisite's poor planning and failure to execute caused this problem. It has caused problems for many clients and many of their client's clients. These problems could be financial damage or it could be a damaged reputation. No one else is to blame except for the failure of a corporation that is paid to entrust and ensure connectivity.
Posted by: vicory at November 7, 2007 5:06 PM
All that you have said and done over this situation should cause YOUR clients to DROP YOU as it is obvious to anyone with a little knowledge of websites and how they work, that you do not have the FIRST CLUE about the business that you are in. The total incompetence that you showed your clients by not having your own backups or even understanding the need for your own independent backups, by not having a contingency plan for server interruptions, by acting like a child who's daddy has abandoned her during this incident, in short by not doing anything you should have done and everything you should not have done, you have proven to everyone that you are incapable of handling even this minor situation, and this was and is a MINOR SITUATION as pertaining to Web based businesses. Grow up and take responsibility for your own mistakes instead of wasting your time ranting and raving insensibly like the immature brat that you seem to be. And get out of any web based affairs until you at least gain a basic understanding of what is needed and how it works.
Posted by: Jean at November 7, 2007 2:20 PM
Don't know exactly what your hosting needs are, but my wife and I run our website from HostGator. We previously worked with GoDaddy and HostNine, but have had server lag issues with GoDaddy and Outages for at least 20 minutes a day on HostNine, and we're still fighting with them over a billing issue.
So far no Problems with HostGator, everything has been extremely fast and smooth with no downtime and they're giving us decent bandwidth for the price.
I'd say to make sure to back up your databases and sql tables often though, that way if you ever do have issues, it makes it very easy to switch. Because of this, each time we switched there was downtime of less than five minutes.Posted by: JeffG at November 7, 2007 2:04 PM
My website came up yesterday for a half of a day for the first time in days, but down again today. We are a small business, but busy. Have lost appro $20K in revenue and $5k in personal income.
Posted by: hanna at November 7, 2007 1:24 PM
My 3 sites are *still* down. At some point on Monday, about 5 emails trickled through. Other than that... nothing.
A big *thank you* to Cynthia for documenting NaviSite's debacle and for getting the attention of the media. A Google News search show articles appearing in the Baltimore Sun and Computerworld regarding this botched move. And congratulations on getting your site back quickly; the squeaky wheel did get the grease.
I hope that you still cover this fiasco until the last of the 165,000 sites affected are back online, and through any legal proceedings that follow.
Posted by: Erik at November 7, 2007 12:09 PM
i own a small web development company that has about 30 websites through Alabanza / Linux Web Host. Most of the sites are back up except for 5, and my company website. My clients cannot get into their control panels. Does anyone know which lawyer will take on this class action suit. if so, sign me up and let me know.
Posted by: Eric C Smith at November 7, 2007 10:11 AM
NaviSite,
Run FROM Us.
Posted by: Joseph at November 7, 2007 10:06 AM
"My website came up 11/6, but it's down again this morning (11/7)."
Same thing here.
NaviSite - Run With From Us
Posted by: Steve at November 7, 2007 8:55 AM
I have 7 sites that were affected by the NaviSite outage. They're back up now, but my hosting provider is actively looking for a new data center.
I've noticed that in the few comments that have come from NaviSite regarding this whole mess, they indicate that the outage has only affected 5000 to 7000 websites. What a complete lie! I've included a link here to their OWN site that announces their acquisition of Alabanza. In it, they state that "Nearly 165,000 active web sites in more than 25 countries are powered by Alabanza."
They definitely know how many sites this prolonged outage has affected.
I just thought you might want to know about the page before NaviSite takes it down.
Good luck!!
Posted by: Dave at November 7, 2007 8:44 AM
My company can provide hosting services as well as design services. If anyone would like to discuss switching to us please feel free to call or email. steve@prowebassociates.com or 717 207 7125 ext 102
Posted by: Steve Maisch at November 7, 2007 8:32 AM
My family business website and email is still missing this morning. I listened in to the conference call at 8:00 PM yesterday. Beware, if your site is up again, because one of the complaints from the web hosts was that each server Navisite gets back up knocks another one down-therefore no real progress is being made.
Good luck to all.
Posted by: jim curley at November 7, 2007 8:32 AM
My website came up 11/6, but it's down again this morning (11/7). FTP hasn't worked since this nightmare began, and E-Mail has only worked sporadically for the last five days.
SLA's don't help when your income has evaporated.
Posted by: Mike at November 7, 2007 8:03 AM
We moved to http://hostspring.com. They honored the money we had already paid in advance to our current host at Alabanza after I explained the situation!
Posted by: Karla at November 7, 2007 7:42 AM
www.meditemple.net
Posted by: Todd Santoro at November 7, 2007 2:48 AM
No, Navisite is not a decent company. Navisite is a bunch of lying sacks that outsource their customer service to India in order to squeeze every last drop of profit out of their suckers, I mean customers, who pay the bills.
They had no regard at all for the meltdown that they would create for thousands of small yet important web hosting providers who's reputations they destroyed overnight.
To top it all off, once the situation was underway and the crisis was at full boil, no Navisite execs came on the many conference calls to apologize and give the real story about why they decided to do what they did and exactly what they were doing to recitify the situation.
They pushed forward well meaning but ultimately powerless customer service reps into the pit instead. How's that for failing to stand up for your own decisions...?
Sorry Rob, these guys suck. They did everything wrong and then lied about it whenever it pleased them. They are not a "decent" company, unless by that you mean "decent into Hell..."
Posted by: TotallyFixed at November 7, 2007 1:30 AM
Steadfast Networks perhaps?
Posted by: Josh at November 6, 2007 7:05 PM
I haven't worked for Navisite for several years. But I can tell you that it's easier to fly to Manchester, New Hampshire and drive south if you want to get to their office in Andover.
Posted by: Rob at November 6, 2007 6:54 PM
My three sites that were hosted by linuxwebhost / Alabanza are still down at of 3:20p on 11/6. My most essential site, www.valicom.com, is reporting a timeout error; perhaps though that's better than the 404 error that had appeared up until recently.
Still can't connect to my mail server there.
I just need my sites in operation long enough to grab a few files from them that I haven't backed up locally and head off to a new hosting service. I'm not giving another dollar to LinuxWebHost, Alabanza, or NaviSite.
BTW, any recommendations on alternative hosting services?
Posted by: Erik at November 6, 2007 6:31 PM
Now begins the damage control. Rob from Navisite Corporate?
Drop them. If not already in the works, host the site yourself or get another provider that actually respects the importance of your business and livelihood.
It seems too often today that these businesses can interrupt and interfere with others livelihood while reaping little or no punishment.
I'm certain the piss-poor planning was the result of an overall cost reduction strategy. As a result, the customer got caught up in the middle of the poor execution.
What make this situation even worse, as I read it, is the amount a contempt shown by Navisite's support to it's customers. Lying, stalling, no real apologizes, not informing customers ahead of time, providing false information, etc, etc...
The brilliance of the management to save money will likely end up costing them more than they could imagine. But only if, the customers -- you -- punish them for their incompetence and the contempt shown to their customers. Otherwise they'll just spin this event out of existence. Good luck.
Posted by: Vicory at November 6, 2007 6:09 PM
Thats what SLA's and redundancy is for :).
Navisite is a decent company.
Posted by: Rob at November 6, 2007 6:04 PM