DESCRIPTION

Add persistence verification check

Will check persistence across all the nodes in parallel, and compute
the 90th percentile value of the ep_flusher_todo metric.
When ran in parallel with load, this value should be non zero.

Change-Id: I24c68f86701c898d8cfff6b3c4d1201bfeec9c41


COMMENTS

author: Reid Stokes
date: 2011-10-08 01:41:40.667000000

Patch Set 1: Verified

Have verified with load running in parallel. Assuming 90th percentile to be a safe value to fail the test. 
Need to add this method to the various tests, eg. rebalance etc.

-------------------------------------
author: Zackery Barr
date: 2011-10-08 02:00:22.330000000

Patch Set 1:

we can also increase the test duration to run for x minutes first before we start validation . 
that makes it easier for us to run few clusters in parallel and monitor the run,

-------------------------------------
author: Ayden Rivas
date: 2011-10-09 04:58:05.372000000

Patch Set 1: Looks good to me, but someone else must approve

(2 inline comments)

overall it looks fine, and at least for now if we need to we can push it through as long as we address the issue of verification being called on a bucket that has all its items already persisted.

Line:108, lib/membase/helper/cluster_helper.py -> the log should show what stat we are getting the percentile in.
perhaps also comment on what this percentile actually means.

Line:110, lib/membase/helper/cluster_helper.py -> this should also check to see if there are actually items that need persisting. if this function is called and all items have already persisted to disk then it will return False, when it seems like it should return True

-------------------------------------
author: Reid Stokes
date: 2011-10-10 03:19:51.678000000

Patch Set 1: Looks good to me, approved

Okay. I will push this change in. 
Will also add the modify the tests to take care of these changes.

-------------------------------------
author: Reid Stokes
date: 2011-10-10 03:19:56.091000000

Change has been successfully cherry-picked as 0b8edc375d8b7e3be9da5a329f6df2704fd1c631.

-------------------------------------
