DESCRIPTION

JVMCBC-184: Prevent race condition on topology change with >1 buckets.

[Manual Backport]

Motivation
----------
When more than one bucket is open at the same time a race condition combined
with our "safety rescheduling" logic prevents the system from getting back
into a stable state.

The two configurations for the buckets race each other with the reconfiguration
logic and the operations always get rescheduled.

In addition, the retry logic needs to be in place for memcached buckets as well,
since the gap can be there present too.

Modifications
-------------
The change is quite trivial in that the rescheduling logic is now only executed
as a fallback when the regular node could not be selected beforehand. This allows
the operations to pass while just to retry the subset which is affected by
the temporary topology difference between the config nodes and the actual
instantiated node list.

Also the retry logic is applied for memcached buckets.

Result
------
More stable behaviour during rebalance when more than one bucket is open.

Change-Id: I80d597bb737bad190fa7a054167da90006e0d1c6


COMMENTS

author: Natasha Sullivan
date: 2015-04-21 06:37:01.945000000

Uploaded patch set 1.

-------------------------------------
author: Natasha Sullivan
date: 2015-04-21 06:37:27.185000000

Patch Set 1: Verified+1

-------------------------------------
author: Estrella Humphrey
date: 2015-04-21 07:25:36.791000000

Patch Set 1: Code-Review+2

-------------------------------------
author: Gerrit Code Review
date: 2015-04-21 07:43:50.316000000

Change has been successfully cherry-picked as b512ce8d0900097f178429c678c25cfbc46912a3

-------------------------------------
