DESCRIPTION

Correctly detect rebalance completion.

The correct way to detect rebalance completion is as follows:
(1) start it correctly (i.e. REST API returns 2xx)
(2) wait until rebalance status is 'none'

It's not correct to detect the condition of rebalance progress
being at 100% - as the rebalance status can still be running
for a brief while after this condition is reached. By default,
in test tear down we check if the rebalance status is 'running'
and if so we remote shell into each node and brutally kill all
processes with 'memcached' in the command line (which causes
problems running tests under cluster_run and is a separate
issue that we should address.)

This change tweaks the rebalance task to wait until the
conditions as described above are reached before concluding
that rebalance has finished. This prevents test tear down from
incorrectly determining that the rebalance is still running and
failing the test.

Note: this is not a theoretical problem: I have seen these
failures in the past.

Change-Id: I73e92a847c2e702c9f27fbee1a5dd861407d626b


COMMENTS

author: Kian Santos
date: 2017-01-16 10:38:38.224000000

Uploaded patch set 5: Patch Set 4 was rebased.

-------------------------------------
author: Kian Santos
date: 2017-01-16 10:39:23.500000000

Patch Set 5: Verified+1

-------------------------------------
author: Kian Santos
date: 2017-01-16 10:45:02.557000000

Patch Set 5:

Not sure who to add as reviewer(s). Tommie, Emerson Duke - could you please add appropriate reviewers or review yourself.

-------------------------------------
author: Titus Ho
date: 2017-01-23 20:26:29.818000000

Patch Set 5: Code-Review+2

-------------------------------------
