flappy node damping

[jwolski]

ringpop needs the ability to detect slow or flappy nodes and intentionally evict them from the cluster.

can we introduce a damping period akin to the suspect period? the period would be initiated if a node within the cluster had to assert itself as an 'alive' member over some yet-to-be-defined threshold. this node can mark itself 'damped' and generate an initial damping score based on the degree of flappiness. that score can then grow or decay based on behavior of the cluster over the damping period window. if the score does not decay enough the 'damped' node will eventually be marked as 'evicted'. evicted nodes must be removed from the ring (or ownership of the keyspace significantly reduced), but may still be part of the cluster membership and thus pinged periodically (but less frequently) as part of the protocol period.

Potential inspiration:

• http://www.ietf.org/rfc/rfc2439.txt
• http://www.potaroo.net/ispcol/2007-06/dampbgp.html