High Availability of a web server on a distributed cloud

[Digimer]

On 01/14/2013 09:54 AM, Dave Cramer wrote:
> 
> On Mon, Jan 14, 2013 at 9:43 AM, Digimer <lists-5ZoueyuiTZiw5LPnMra/2Q at public.gmane.org
> <mailto:lists-5ZoueyuiTZiw5LPnMra/2Q at public.gmane.org>> wrote:
> 
>     On 01/14/2013 09:33 AM, Jamon Camisso wrote:
>     > On 13-01-14 08:57 AM, Dave Cramer wrote:
>     >> It is relatively simple to distribute an application to a number
>     of servers
>     >> and use haproxy to switch ip's. What I can't figure out is how to
>     switch
>     >> make sure that the IP that points to ha-proxy can be moved easily
>     if that
>     >> machine fails ?
>     >
>     > Research things like pacemaker, heartbeat, stonith, and using 3 or
>     more
>     > nodes in your system (3 nodes so that you don't end up in a split
>     brain
>     > situation, especially if your systems are distributed).
>     >
>     > You'll likely want another back-channel method of communicating
>     between
>     > each system in case something goes awry - what if one your ISP/cloud
>     > providers can't route public traffic to a node, but the node still
>     > thinks it is online?
>     >
>     > You want some method of achieving a quorum between remaining
>     nodes, and
>     > some method of killing off the rogue node.
> 
>     A few points;
> 
>     - Heartbeat is deprecated, don't use it.
>     - Quorum is optional. Both pacemaker and cman/rgmanager (Red Hat) use
>     corosync for cluster communication and membership. Both can disable
>     quorum and this is fine.
>     - The members need to be (physically) close to each so that you can use
>     fencing. Fencing is critical to safe operation of a cluster. Without it,
>     it is easy/possible for a split-brain to occur.
> 
> 
> 
> If you place your machines physically close to one another then a
> geographical outage can take them both out. I guess pacemaker is more
> for machines dying than for geographical high availability 
> 
> Dave

That's called a "stretch cluster" and they are very hard/expensive to
build properly. In order for a service to fail over without risking a
split-brain, the failed node(s) has to be put into a known state, which
is what fencing does, by (usually) forcing it to power off. Even if you
build in dual links between the buildings, you will have to run them
over physically different links in order to avoid, for example, a dig
cutting both at the same time. Even then, a fire or similar disaster can
easily take out both links near the end-points.

What I tend to recommend to/do for my clients is to setup production
clusters for day to day use. These provide fault tolerance and HA
without lost data in the case of component failure. This protects them
against the normal failure modes that cause outages. Then, if the client
is interested, I will create a geographically disparate cluster with
enough VMs to run things in a degraded state in the event of a local
disaster.

The client then runs hourly, quarterly or daily "traditional" backups to
the remote site's VMs. If the production cluster(s) are disabled, they
can switch over to the backups by simply having them take over the IPs
of the lost machines. Yes, in this case some data will be lost, but the
costs are _much_ lower and the risk of split-brain is gone. However,
failover is purely manual. In most cases, clients consider this to be a
justified risk/hassle, given the conditions under which it will be needed.

In short; automatic failover requires fencing to be safe. Fencing
requires independent confirmation of a node's state. That requires a
working network connection and a working fence device on either end.
Anything in the same place as the target node is vulnerable to the same
fate as the nodes themselves. Esoteric fence methods may be possible,
but they would likely require some serious costs for moderately more
reliable operation.

-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
--
The Toronto Linux Users Group.      Meetings: http://gtalug.org/
TLUG requests: Linux topics, No HTML, wrap text below 80 columns
How to UNSUBSCRIBE: http://gtalug.org/wiki/Mailing_lists