[dev] Chris: The Redundancy Plan

[dev at lists.powerblogs.com]

Posted by Chris:
The Redundancy Plan
http://dev.powerblogs.com/posts/1126062861.shtml


   Ok, here's the plan. It's ambitious but achievable, and worth it. (And
   it shouldn't drive prices up, either.)

   Servers will work in pairs, though the implementation will allow for
   trios, etc. They will both serve web pages as well as take new posts,
   comments, etc. The servers will automatically sync with each other in
   real-time on a peer-to-peer basis, so they'll be capable of operating
   independently if one of them goes down or they get separated, and
   they'll automatically sync up as soon as they can talk to each other
   again. (Each server will have a sync daemon which stores all updates
   with a revision number; when a server reconnects it will just ask for
   all udpates from the last revision it knew about -- this will ensure
   smooth resyncing.)

   Reports will be generated by an off-site server, taking the load off
   of the web servers to ensure that report generation will never bog
   them down. (I'll have to get a feel for how quickly I'll be able to
   process the reports, but it might be reasonable to get the processing
   up to every 2 hours, which would be really nice.)

   All of the servers will be DNS slaves (listed in the master record) so
   that there won't be any single point of failure. DNS will hit the two
   servers in round-robin fashion.

   As I said, it's ambitious, but it will result in a system with no
   single point of failure that should be able to incorporate new servers
   quickly and seamlessly. Getting this done is going to be top priority
   here at Powerblogs. The transition to the two redundant servers should
   be completed within two weeks.

   I'll keep you updated.