DESCRIPTION

JVMCBC-111: Fall back to other cluster nodes on carrier refresh.

Motivation
----------
In the current codebase, only the first node in the list was asked for a valid
configuration. This leads to problems when exactly this node is failed over
or removed. As an effect, no valid configuration can be fetched anymore, which
in turn keeps the removed node from properly being ejected.

Modifications
-------------
The carrier refesher code has been refactored so that the nodes are tried in order,
and if one node is not returning a proper config, the others are tried afterwards.
This makes sure that even when a node is being removed or failing, another one is
tried eventually, bringing the system back to stable state.

Tests have been added to cover this scenario. Note that on other tests the wait/grace
time has been increased because the new interval method which is used really starts
after 1 second and not immediately.

In addition, the code also shuffles the node list to reduce the probability that the
same (unavailable) node is hit over and over again, making it quicker to recover.

Result
------
Much better and stable config management when the first node of the cluster list
is being removed against 2.5.0+ nodes.

Change-Id: I95cce02c54b63da177e84f8eb178bcd7eadc9bbc


COMMENTS

author: Natasha Sullivan
date: 2015-01-02 13:51:18.920000000

Uploaded patch set 3.

-------------------------------------
author: Natasha Sullivan
date: 2015-01-02 13:53:19.965000000

Patch Set 3: Verified+1

-------------------------------------
author: Estrella Humphrey
date: 2015-01-02 13:57:27.076000000

Patch Set 3: Code-Review+2

-------------------------------------
