|
MstrssPassion -> RE: Wondering... Are you here tonight because B.com has crashed? (6/4/2007 5:01:54 AM)
|
not sure why it isn't working for you this is what I get...
(edited to remove pictures)
**********************************
Posted: 2007-06-03 23:59
The Amazing Spiderman
Administrator
United Kingdom
1,276 Posts
In order to keep everyone updated as to whats going on, I've posted this brief service status post, its locked to keep it from getting out of control. I'll keep it updated when we get any large scale problems. The various states are
[* RED] - Serious Problem [* ORANGE] - Testing/Minor Issue [* Green] - No Known Issues
Service Status
[* RED] Videos (Free List) Feed link errors
[* RED] Site Hardware failure
[* Green] IRC No Problems Reported
[* Orange] Email Problems confirming email addresses
[* Green] AVChat No Problems Reported
Service Updates
Since posting my last update, the database server died again so had to be re-restored. This meant my status update dissapeared.
We appear to be having multiple harware failures on our database server, we are currently building more new ones, and transferring data over from it.
I apologize at the moment I can't answer/read all the forum feedbacks since I'm not at home I'm currently on holiday and doing all this over a cellphone. I've edited some code to make blogs/emails/pictures/forum/user bios work without the secondary database we have but searching is down.
We are also flying in people from all over the country to palo alto to try and get this problem sorted ASAP, but I understand its frustrating (It is for me too since I can't really give any answers or explanations for it! All I can say is sorry and people are working on it! Something fucked up somewhere and we are doing everything we possibly can to get it sorted, and there will no doubt be more contingency plans made in the future and I believe we are going to make more reundent machines (ie instead of just having a redundant machine we will have redundant machines for the redundant machines).
We are also not censoring other threads (i seen some headers about free speech on bondage.com) we simply lost the db and had to restore from backup (this meant any threads since the last backup were lost).
I am keeping up to date on the issue, and checking in as much as possible, it is a hardware issue so I Can't really comment much more (I'm not a hardware guy) but I've changed the software as much as i can to get as much features back up as possible, I Realise AVChat is down, so i would recommend using the IRC servers at the moment (it is a seperate entity so will work).
Mitch
Upcoming Maintance
No Maintance Outages Planned
--
___________________________________________
mitch
Posted: 2007-06-04 00:26
The Amazing Spiderman
Administrator
United Kingdom
1,276 Posts
June 4th - 00:21
Hey,
Ok, I've finally got net access on a computer and have been looking through all the posts and looking at all the emails I've got.
First off, I realize this is frustrating, and everyones pissed off but please be assured you are not the only ones! The technicians are working 35+ hour shifts. The support people have canceled their weekends, the executives are no doubt popping veins in their foreheads and I'm 1700 miles from home in a lovely sunny beach city.... spending my time indoors on cellphones and laptops (with a sunburn!) No one could of seen this coming, but what we need to do now, is get it usable first, get it fixed properly, find out how to move forward ie release details of what we are going to do to kill members concerns and then plan how to avoid this in the future.
For the technically challanged heres a quick recap :-
The site works off a database server which is just a another computer which stores all the sites data on it, ie forum posts, user accounts, password etc etc. Inside that database server we have a unit called a RAID controller, now the purpose of that device is we can plug in two hard drives, and if one hard drive was to break, the RAID controller would detect it and copy data from the good hard drive onto the broken one to fix it. Now heres where the problem is, the RAID controller (Which is designed to stop breaks) broke itself, that means it was effectivly breaking the hard drives. This meant we had to goto the backups and get a copy of stuff we had saved at a earlier date and put it on the hard drives.
For the technically able:-
The SQL Servers RAID controller popped one of its channels, the SQL server is not currently clustered because of the schema of the database it makes it very difficult/impossible to use clustering (yes there are times clustering accually causes more problems than solves!). We are in the process of building some new SQL servers, so we can pop the data on them and run off them on newer faster (more reliable) hardware.
All together now:-
I'm going to keep this post locked, so I can add new updates when I get them, and for reference I've kept the old service status thread around (just locked it) but this is the one you should be looking at! I realize your all eager to voice your opinions and concerns about the issue and that is understandable, you can do it at this thread but please please please avoid insults and stuff. Even though the site has currently gone 'tits up' the AUP still has to stand so we don't get some riots!
For the next couple of hours I'm going to be working on some back end code to see if i can improve the situation a little more and make things a little more stable and then I'll grab a couple of hours sleep and start answering forum posts again tomorrow morning. In the mean time theres some technicians currently getting parts together and building new servers, installing MSSQL on them and getting the data over there (so theres still work being done).
So while theres is work being done, just find your sub and beat them, or ask your Dom(me) for a beating to pass the time. And try not to get too stressed (I'll do enough for us both!)
Thanks
mitch
|
|
|
|
