DESCRIPTION

JVMCBC-111: Fall back to other cluster nodes on carrier refresh.

Motivation
----------
In the current codebase, only the first node in the list was asked for a valid
configuration. This leads to problems when exactly this node is failed over
or removed. As an effect, no valid configuration can be fetched anymore, which
in turn keeps the removed node from properly being ejected.

Modifications
-------------
The carrier refesher code has been refactored so that the nodes are tried in order,
and if one node is not returning a proper config, the others are tried afterwards.
This makes sure that even when a node is being removed or failing, another one is
tried eventually, bringing the system back to stable state.

Tests have been added to cover this scenario. Note that on other tests the wait/grace
time has been increased because the new interval method which is used really starts
after 1 second and not immediately.

In addition, the code also shuffles the node list to reduce the probability that the
same (unavailable) node is hit over and over again, making it quicker to recover.

Result
------
Much better and stable config management when the first node of the cluster list
is being removed against 2.5.0+ nodes.

Change-Id: I95cce02c54b63da177e84f8eb178bcd7eadc9bbc
Reviewed-on: http://review.couchbase.org/44837
Tested-by: Natasha Sullivan xxx@xxx.xxx
Reviewed-by: Estrella Humphrey xxx@xxx.xxx
Reviewed-on: http://review.couchbase.org/44911
Reviewed-by: Natasha Sullivan xxx@xxx.xxx
Tested-by: Estrella Humphrey xxx@xxx.xxx


COMMENTS

author: Estrella Humphrey
date: 2015-01-05 19:15:38.339000000

Change has been successfully cherry-picked as b45110825c5454a4c4e04bda8276ded45616644d

-------------------------------------
