Applications services and managed hosting provider Navisite is in the midst of a massive outage as the publicly traded company attempts to consolidate its data centers. Unfortunately, my sites (not this blog) and email have been down for the count going on two three days, with no end in sight.
For the past two three days, Navisite has been promising to fix the issue within a "few hours." Now they are promising to resolve the problem by midnight [ed. midnight deadline came and went with no resolution], although at this point very few clients trust what the company has to say.
Navisite has hosted several conference calls promising a solution to the unexplained problem, but none has materialized.
At least 1,400 organizations host their servers with Navisite and at least some of those servers are partitioned among multiple parties. One commenter (link here but the site is down as of 10:03 a.m.) has suggested that at least 200,000 web sites have been pulled down by Navisite's poor planning.
One lesson painfully learned from this episode: if you can avoid it all, don't outsource your hosting. And if you're not in a position to bring the hosting in house, don't, under any circumstances, rely on Navisite to get the job done.
Update: As of 7:40 a.m. this morning, Navisite has still not resolved the problems that have shut down several hundred thousand web sites. The company's PR contact phone number is "not in service" and I just spoke with a Navisite technical support representative who claims not to know about the source of the problem. He promised to have someone call me. An estimated 175,000 sites have been knocked offline for the third day in a row, most of them are either small mom and pop operations, the lifeblood of the Internet, or non-profit agencies. A good diary of the screw-up is here. Mobile Mind recommends finding another managed hosting provider.
Update: If you're a Navisite customer and have been down for the past three days, please post a comment and tell me your story. It's inconceivable to me that so many sites have gone dark and the mainstream press hasn't picked up on what's going on with Navisite.
Update: Three days following the migration disaster, Navisite has gone public with a statement. They've promised to have a "significant majority" of us up and running by 1 pm. EST today. (Of course, they've been promising to have everybody up and running all weekend.) Still amazing to me that the mainstream press isn't all over this.
Update: Guess what? The 1 p.m. deadline came and went my sites are still down. We're definitely going to ditch Navisite when this is all over and I hope they compensate us for the time and cost it will take to do so.
Cynthia Brumfield at 9:53 PM|Comments(18)
We were down from Friday night to Tuesday night. There was 2 hours on Monday we were up. All our clients were also down. Obviously we are in process of moving servers.
Posted by: Noel at November 10, 2007 5:39 PM
Our company InfoLink IT's Servers have been down since Saturday.
InfoLink IT's Director is commencing legal action against NaviSite for company and client compensation and after 30+ phone calls and no return phone calls we were HAD TO TELL NaviSite what they need to do to get our servers up and running.
It is very very very sad when our techs .. where we are a company of only 6 people had to tell a publicly listed company that has over 500 staff how to fix a major catastrophe.
If you are in the same boat as us, YOU NEED TO TELL THEM TO REBOOT THE APACHE ON THE SERVERS. They did not believe that this would help, but to their surprise, it got all 800+ sites that were down live again.
We had 3 servers... one working that only held 10 sites and 2 down, that held over 800...we have been trying to request the root password for 12hrs which they could not give us because they told me that the tool they use to retrieve this password is BROKEN.
We were very persistent and have been on the phone for over 5hrs today and told them that all they need to do is restart the apache on the servers.. finally after getting escalated to a higher tech level and talking to every Indian in their call center we get onto someone that would actually do what we wanted.
Within 10 seconds of them restarting the apache ALL OUR SITES WERE LIVE AGAIN!
We hope this info helps you get your sites live again
and as our director says.. it takes Aussies once again to come in and bail out the Americans!
InfoLink IT
www.infolinkit.com.au
If anyone would like to join the class action lawsuit, please contact Paul Klerck Director at paul@infolinkit.com.au. It is headed up by Minter Ellison Lawyers Australia.
Posted by: Paul at November 7, 2007 1:57 AM
Just received an email and request from a navisite customer explaining the situation. and asking for us to possibly host some sites for them. They sent me this link to offer our services through this blog and to help where we can.
I work for afull service ISP/Communications company in Buffalo NY with redundancy in another part of the state (Rochester). We hate to see anybody down for any time but three or more days is awful and would be glad to do what we can for anybody who can not afford to be down any longer.
While this will benefit me and the company I work for, it will also benefit those of you who are down. We should be able to move your service within a few hours (assuming we can get navisite to do their part). Contact me. We will do everything we can to help. You can contact us at: 1-866-295-7724 or 716-881-6132 or mmosey@winccommunications.com
Reference this situation (Navisite down)
Posted by: Michael at November 6, 2007 7:22 PM
We started knowing about issues when we couldn't connect to our mail or website every few hours or so for several minutes at a time last week. Monday when it became clear they were playing around last week before their move, which we took as a precurser to them not knowing what they were doing, we decided to jump ship. Luckily we had a backup of the whole back end on our server. We picked a Network Solutions package and loaded the website within an hour or so. Tested it on the back end for another hour and then pulled the trigger for the domain move (which was already with NS). That was done around noon, Monday. By 8:30PM that night we had full connections to the web and mail. After a few changes in everyone's Outlook, we began receiving mail after a few minutes. All told about 10-11 hours after our decision we were back in business.
I too can not belive that the media has not picked up on this....yet.
Posted by: Mike B at November 6, 2007 4:11 PM
Yes...the nightmare continues. 112 HOURS of downtime to date. We have 5 servers that were with Alabanza (now Navisite). We've sent numerous emails to nocmonitor@navisite.com as directed. I've called their support line 5 times to be put on hold for 30 minutes plus and disconnected each time. The frustration is unbelievable. Our company has lost three major resellers and an untold number of single domains we host. The website updates are a joke with their constant...two more hours...for 112 hours... then in the midst of all of this I get my monthly INVOI CE this morning! What???
I believe a class action lawsuit may get their attention.
Posted by: Colleen at November 6, 2007 3:40 PM
Hello
We are a Canadian company situated in the middle east and have more than 200 websites hosted previously with alabanza. I propose to do a lawsuit to have us and all those in the same disaster indemnized.
our lawyers are working on a case as we speak, could you please contact me at: fschoucair@elementc2.com should you have any comment and concerns, or would like to join.
faridsch@hotmail.com
best regards
F. Schoucair
Posted by: Farid Schoucair at November 6, 2007 3:24 PM
Here's my online chat with a NaviSite rep, as we go into the fourth day of my three sites being down (originally hosted by Linuxwebhost / Alabanza). Note how NaviSite's rep Barbara's responsive are just about as useful as the info on NaviSite's website.
System: Please wait while connecting to an operator.
You're connected to an operator - Barbara
Barbara: Hello! How can help you?
User: Hi Barbara, I have 3 sites hosted by Linuxwebhost/Alabanza that are still down.
User: Can you give me any estimate when I will get email and web services back?
Barbara: I don't have that information
Barbara: For the latest update you can go to our website at: http://www.navisite.com/sublevel.aspx?id=2017 or contact customer care at: customercare@navisite.com
Barbara: We have everyone available working to resolve this issue. We certainly appreciate your patience in this matter
User: Just a moment, let me check that link. Thanks.
User: That link gives me a response that doesn't have any timeframe for the sites to come back up. Downtime associated with this move was supposed to be Saturday only. It's now Tuesday. Who specifically can I contact to get an answer about when my hosted sites will be available?
Barbara: I don't have an answer to that ... you can continue to check the website for updates
Barbara: We have everyone available working to resolve this issue. We certainly appreciate your patience in this matter
User: Please give me a specific name with phone number or email address. The website updates aren't helpful or specific. In the meantime, my business is suffering. I can't even tell my clients how long an outage to expect.
Barbara: I don't have a specific name or phone number to give you
Barbara: We have everyone available working to resolve this issue. We certainly appreciate your patience in this matter
User: Where, then, can I get a better assessment of the situation than "We have everyone available working to resolve this issue. We certainly appreciate your patience in this matter," and the similarly unspecific statement on your referenced website?
User: There's no indication of whether the downtime will be an hour, a day, a week, a month. I need to know, as do my customers.
Barbara: I don't have that answer...I can only tell you that we are working on it
User: So the downtime could be a month?
Barbara: I would hope not but I don't have specific information to the timeframe
Barbara: We have everyone available working to resolve this issue. We certainly appreciate your patience in this matter
User: Ah, I see.
Posted by: Erik at November 6, 2007 2:18 PM
This must have been the biggest hosting disaster ever...
my site is down for the 3 day in a raw, I am loosing business and reliability to my customers...
I cannot overstate how angry I am. The this fiasco is over I am moving our site incompany.
Posted by: J Stam at November 6, 2007 7:27 AM
For small businesses, it's like watching the WTC buildings come down. You don't think it can get any worse, and then it does. By the way, Navisite's (NAVI) stock went up today. I guess Wall Street doesn't know what's happening with the company's customers, many of whom are probably looking elsewhere for a new web host.
Posted by: Richard McCormack at November 5, 2007 5:19 PM
Click-To-Chat Service at Navisite (the following is the conversation between myself and "Ed" an operator at Navisite. Dated: 11/05/2007 @ 4:47pm -
System: Please wait while connecting to an operator.
You're connected to an operator - Ed
User: Hello Ed
User: I am Gregg V
Ed: Hello. Can I help you?
User: If possible, can you determine the ETA of when the south Florida website servers will be back online?
Ed: Hang on, please....
User: ty
Ed: Gregg, for the latest information on datacenter migration please go to the following link on our website: http://www.navisite.com/alabanza.aspx
Ed: The latest information is there.
User: I have already done so.
User: That issued message was posted @ 2:35pm EST and the article stated 2 hours from then.
User: It is now 4:47pm EST. That is why I am attempting to gain further info.
User: Are you there?
Ed: That is the latest information we have as well. I know they are working as fast as possible to have all issues resolved today, but I don't have any more info on a timetable, I'm afraid.
User: Ok, thank you. Good Luck!
Ed: Thanks for your patience!
System: The Operator has closed your chat.
Posted by: Gregg V at November 5, 2007 4:51 PM
Alabanza and NaviSite have been down amost 70 hours and counting! WOW! It is clear at this point that most of Alabanza's clients are already out of business, I can not imagine anyone who would suffer a 70 hour outage and not 'jump ships'.
NaviSite have stated that it could be another 12 hours. You can access their conference calls here:
877-720-6284 - 2064734
Posted by: DigiHost at November 5, 2007 4:34 PM
I first noticed email problems on the 31st, so this screwup has been going on longer than they are admitting. My hosting company is Earthplaza, and they are scrambling with at least hundreds of sites down, including mine, which include non-profits and small businesses. Email sent to those customers during this outage is gone forever, so who knows how much is lost. I have one site back up, and Pam at Earthplaza just told me she was on another conference call with nothing definite. The Navisite web site continues to misinform. The small hosts had no fallback- they were told about the migration, but when the data transfer failed Navisite didn't stop, and without telling anyone they unplugged the servers and threw them in trucks for the weekend. People were ripped off the internet with no warning. Navisite will probably next try to blame their customers.
Posted by: Michael H. Bartlett at November 5, 2007 4:08 PM
3pm Monday, still down.
Posted by: Lee Byrd at November 5, 2007 3:03 PM
As the owner of a two very stable web hosting companies, I would ask the IT community to please consider more than just price when selecting a web hosting firm. It frustrates me that we see things like this, when there are 100% uptime providers out there for those that can spend just a few dollars more a month for the great service...
Posted by: Jim Salviski at November 5, 2007 2:17 PM
As a former Project Manager at Atlantis Paradise Island (Bahamas) and now a private webmaster, I find it very disturbing and lacking in professionalism that a publicly traded company would take such a risk wherein thousands of website customers (personal and/or business) were affected for days. I urge a full investigation be performed by an impartial commission to determine why Navisite's CTO and Senior Project Management would allow the relocation of servers without proper back-up/fail-over procedures in place.
Considering the availability of technology systems (outsource companies) in today's industry there is no excuse for this outage. Budgets should have allowed for back-up systems to be in place to deter this failure in the project.
I hold the CTO and it's upper management liable for this inexcusable failure in judgement. And I personally am urging my customers to seek alternative website server companies. This is nothing but an 'outrage' and a direct reflection on Navisite's lack of regard for its customers and shareholders alike.
Posted by: Gregg V at November 5, 2007 2:02 PM
They're starting to see progress. We're able to FTP into our site and get email, but still can't access our site via a browser. Our data is still not accessible, but I'm hopeful that it will be available soon. I have to give kudos to Rob at Navisite - he's been a real trooper.
Darrell C. Greenhouse
Here’s a partial transcript of an IM session with Navisite’s point man during this trying period.
(Actual IM names have been obfuscated)
Although I hadn’t saved the beginning of the IM session, the gist was me asking Rob if host machines that our site resides on had been brought up yet…
Session Start (dg-IM:rob-IM): Mon Nov 05 09:04:56 2007
[09:05] rob-IM: All servers have been racked and cabled.., confirmed!
[09:05] dg-IM: That's great! I hope our server comes up! We're dead in the water like everyone else.
[09:19] dg-IM: Well, I can now FTP into our site. So there's progress. Yippee!
[09:19] rob-IM: :)
[09:20] dg-IM: Still can't route to our index page or any pages using the IP address. (I'm FTP'ng using the IP). Very interesting.
[09:36] dg-IM: Back down again. I guess they're bringing up hosts, testing and dropping them either due to too much traffic or other problems
[09:37] rob-IM: They are trying to develop a way to programmatically force a sync of the DSM name servers with the Alabanza name servers
[09:37] dg-IM: Got it
[09:38] rob-IM: I'm going to be leaving for a bit because I've been up for 37 hours without sleep
[09:38] rob-IM: But I will be back
[09:38] dg-IM: Have sweet dreams of digital girls :-)
[09:38] rob-IM: Thanks Darrell!
[09:51] *** "rob-IM" signed off at Mon Nov 05 09:51:13 2007.
Session Close (rob-IM): Mon Nov 05 09:56:34 2007
Posted by: Darrell Greenhouse at November 5, 2007 1:49 PM
Give them credit for talking about it? They are responsible for inflicting untold costs on thousands of small businesses, and they disconnected (or never had) an operational media contact line.
For three days they haven't told a straight story about when the problems will be resolved.
BTW, not 200,000 servers, but 200,000 sites have been affected.
Posted by: Cynthia Brumfield at November 5, 2007 12:29 PM
NaviSite.com has more information. Highly unlikely that 200,000 + servers are down.
I give them credit for talking about it.
Posted by: Kevin at November 5, 2007 12:09 PM
Blogs We Read
