DESCRIPTION

JVMCBC-102: Perform initial bootstrap vertically, not horizontally.

Motivation
----------
Various edge cases in the current bootstrapping process have highlighted
a limitation of the current approach:

Rather trying carrier publication and http loading on each node separately
and picking the first good config that is returned, the current code tries
carrier on all seed nodes and if this fails tries http on all of them.

While this works fine for most of the cases, it falls short when some
of the nodes return errors and others don't return at all (while some
still return good responses!).

Modifications
-------------
The current loader code uses mergeDelayError which doesn't work well under
the situations described above. Sometimes the code waits too long, while
under other circumstances it doesn't wait until the good message comes back.

Looking at this approach from a more high level view, it turns out that not
either cccp or http is failing, but rather a specific node is either available
or not.

Realizing this, the code has been modified to now run the individual loader
chains for each seeed host individually, merging it at the very end and picking
the first good one.

So even if one node responds with an error or doesn't respond at all, a good
node has the chance to send a config back to the bucket manager, leading
to a finished bootstrap process.

Tests have been modified to reflect these changes, also the Loader API has been
modified to only take a single seed node instead of a set and of course now
also doesn't perform merging and taking at this level.

Result
------
The bootstrap process is now much more reliable to individual node failures.

Change-Id: Id3423de96f2545be61397078f9ca4532edb282ab


COMMENTS

author: Natasha Sullivan
date: 2015-11-05 09:15:34.631000000

Uploaded patch set 5.

-------------------------------------
author: Gerrit Code Review
date: 2015-11-05 09:41:49.452000000

Change has been successfully cherry-picked as 4e9ed79243460b9f8367153bcdd8c6f1558b1a26

-------------------------------------
