DESCRIPTION

CBQE-2187 Add Verification Library for Rebalance and Failover Tests

Change-Id: Id44cd1d66731542f6572f0d1925fbf1df346c71f


COMMENTS

author: Quentin Ali
date: 2014-04-11 07:27:53.982000000

Patch Set 11:

Ashvinder, based on your comment for more detailed logical comparison, I have added the concept of comparison maps.

A comparison map for a given field defines ‘operation’ and ‘type’

type defines to what value it will be case. Can be {int, float, double, string}

the operation can be logical (>=, <=,<,>,==,!=) or filter which says do not test

Example: {“uuid”:{“operation”:”>=“,”type”:”int”}

Whenever we do not define comparisonMap, the operations are not performed, we only check for equality of values by assuming they are string.

By using the comparison map, one can define different logical comparison on different fields.

-------------------------------------
author: Hugo Blankenship
date: 2014-04-11 08:08:01.935000000

Patch Set 11:

Build Started http://factory.couchbase.com/job/testrunner-Jasmin Rangel-master/1012/

-------------------------------------
author: Hugo Blankenship
date: 2014-04-11 08:48:41.158000000

Patch Set 11: Verified-1

Build Failed 

http://factory.couchbase.com/job/testrunner-Jasmin Rangel-master/1012/ : FAILURE

-------------------------------------
author: Titus Ho
date: 2014-04-11 20:14:36.658000000

Patch Set 11:

(7 comments)

thanks Demetrius Mack for the example!

few more comments.
+ another part i'm very interested in is the use of cbtransfer for dataverification.  I'm concerned it's always pointing to localhost.

Line:164, pytests/failover/failovertests.py -> looks like syntax error

Line:167, pytests/failover/failovertests.py -> a bit trivial, but you can consolidate logic here wihout if/else:

servers = (_servers_, self.servers)[self.add_back_flag]

self.compare_vbucket_seqnos(prev_vbucket_stats,servers,self.buckets)	         self.compare_failovers_logs(prev_failovers_stats,servers,self.buckets)

Line:171, pytests/failover/failovertests.py -> w/spc

Line:58, pytests/failover/failoverbasetests.py -> here you can call self.get_failover_logs() above,

Line:60, pytests/failover/failoverbasetests.py -> everything here is set to False.  this makes it impossible to dynamically set these params externally.  also if they were variables passed into function with default=False then what they represent would be more readable.  ie

compare_failovers_logs(.... addedItems = False, DeletedItems = False ..)

Line:64, pytests/failover/failoverbasetests.py -> False, True, False...  hardcoded

Line:68, pytests/failover/failoverbasetests.py -> call self.get_vbucket_seqnos

-------------------------------------
author: Titus Ho
date: 2014-04-11 20:16:18.928000000

Patch Set 11:

(2 comments)

Line:58, pytests/failover/failoverbasetests.py -> discard...

Line:68, pytests/failover/failoverbasetests.py -> discard...

-------------------------------------
author: Quentin Ali
date: 2014-04-11 22:02:21.036000000

Patch Set 12: Patch Set 11 was rebased

-------------------------------------
