DESCRIPTION

JCBC-681: Detect missing noop polls and rebootstrap config.

Motivation
----------
If the node where the carrier config is attached to, dies silently,
the config channel is kept open longer than its doing any good. We
are already doing NOOP polls on a 5 second interval to make sure
the connection does not get dropped by firewalls, so we can piggyback
the responses to trigger an outdated config signal when they do
not come back anymore.

Modifications
-------------
The changeset piggybacks on the NOOP interval which is done anyways
and adds a simple counting mechanism. When more than (by default)
3 NOOPs are missed, after 3*5seconds a "outdated config" signal is
issued to the configuration provider.

Once the provider tries to grab a config but can't (since it won't
return him one on the dead node), a full rebootstrap is initiated.

Result
------
The client now switches over to another node in the cluster even
before auto failover kicks in, so when it does it can very quickly
retrieve a valid configuration and get back to a stable state.

Change-Id: I7004d1923bf43cf899113da48dd8aa146a458625
Reviewed-on: http://review.couchbase.org/45214
Reviewed-by: Estrella Humphrey xxx@xxx.xxx
Tested-by: Natasha Sullivan xxx@xxx.xxx


COMMENTS

author: Natasha Sullivan
date: 2015-01-19 08:16:31.587000000

Change has been successfully cherry-picked as 14bd22c00565e0018d2fe5db975f7c1d0b20fc35

-------------------------------------
