DESCRIPTION

Correctly detect rebalance completion.

The correct way to detect rebalance completion is as follows:
(1) start it correctly (i.e. REST API returns 2xx)
(2) wait until rebalance status is 'none'

It's not correct to detect the condition of rebalance progress
being at 100% - as the rebalance status can still be running
for a brief while after this condition is reached. By default,
in test tear down we check if the rebalance status is 'running'
and if so we remote shell into each node and brutally kill all
processes with 'memcached' in the command line (which causes
problems running tests under cluster_run and is a separate
issue that we should address.)

This change tweaks the rebalance task to wait until the
conditions as described above are reached before concluding
that rebalance has finished. This prevents test tear down from
incorrectly determining that the rebalance is still running and
failing the test.

Note: this is not a theoretical problem: I have seen these
failures in the past.

Change-Id: I73e92a847c2e702c9f27fbee1a5dd861407d626b
Reviewed-on: http://review.couchbase.org/72028
Reviewed-by: Titus Ho <Titus xxx@xxx.xxx
Tested-by: Kian Santos xxx@xxx.xxx


COMMENTS

author: Kian Santos
date: 2017-02-23 18:11:08.263000000

Change has been successfully cherry-picked as 71bd768cfa24224a13ade2778ce5ffd96d12874f by Kian Santos

-------------------------------------
