DESCRIPTION

CBQE-2187 Add Verification Library for Rebalance and Failover Tests

Change-Id: Id44cd1d66731542f6572f0d1925fbf1df346c71f


COMMENTS

author: Quentin Ali
date: 2014-04-07 23:41:48.426000000

Uploaded patch set 2.

-------------------------------------
author: Quentin Ali
date: 2014-04-07 23:46:33.270000000

Patch Set 2:

Currently, data_analysis_helper.py is not being used in any test case. But will add it soon to rebalance and failover tests.

-------------------------------------
author: Hugo Blankenship
date: 2014-04-08 00:34:03.970000000

Patch Set 2:

Build Started http://factory.couchbase.com/job/testrunner-Jasmin Rangel-master/971/

-------------------------------------
author: Hugo Blankenship
date: 2014-04-08 01:01:55.254000000

Patch Set 2: Verified+1

Build Successful 

http://factory.couchbase.com/job/testrunner-Jasmin Rangel-master/971/ : SUCCESS

-------------------------------------
author: Titus Ho
date: 2014-04-08 15:31:03.448000000

Patch Set 2: Code-Review-1

(10 comments)

See comments inline+
I wasn't sure where a lot of these static methods are being called from or how they are being used.  Would be good to see an example of this verification added to a test.  

Also note we do have strong kv verification that exists using our kv_store which will detect missing keys and tricky cases like expiry keys.
But this is good data integrity checking.

Line:1930, lib/remote/remote_util.py -> source=localhost:8091 looks too hardcoded.  i.e. can cbtransfer be run from hosts beside localhost?

Line:1935, lib/remote/remote_util.py -> Administrator/password will fail on clusters with different auth, pass this in as default values to method here.  i.e.

cbtransfer_memory(self, buckets, user='Administrator', password='password'

Line:1954, lib/remote/remote_util.py -> Does this also support remote host?  I guess my concern here is what happens when I want to run verification from a test machine separate from my cluster?

Line:5, lib/couchbase/data_analysis_helper.py -> these can be global, not necessarily class variables

Line:13, lib/couchbase/data_analysis_helper.py -> spc

Line:17, lib/couchbase/data_analysis_helper.py -> what is the motivation behind not using 'self' and all the static methods?

Line:253, lib/couchbase/data_analysis_helper.py -> recommend set intersection:

missmatch = list(set(info1.keys())  ^ set(info2.keys()))

* gives you list of keys missing from 1 set or another
presentInMap1Only = [key for key in missmatch if key in info1.keys()]
presentInMap2Only = [key for key in missmatch if key in info2.keys()]

Line:302, lib/couchbase/data_analysis_helper.py -> here you're looping over keys in map, but the check is incrementing the index of the value… looks to me like.

doc1 = {'a' : 'val1,val2,val3',
            'b'  : 'val1',val2,'val3'}
doc2 = {'a' : 'val1,val2,val3',
            'b'  : 'val1',val2,'val3'}

comparison does

doc1:a:val1  == doc2:a:val1

doc1:b:val2  == doc2:b:val2

where I 'think' you want to do all the values of each vkey.

*note if that's what you want to do this does imply this verification can only run on a certain type of dataset

Line:390, lib/couchbase/data_analysis_helper.py -> you do not have to do ssh to collect cbstats.  if you know the remote host ip and port you can create a lib_mc_bin client itself and call stats('vbucket').

Line:430, lib/couchbase/data_analysis_helper.py -> same thing here, cleaner to call remote memcached with a client

-------------------------------------
author: Titus Ho
date: 2014-04-08 15:34:44.059000000

Patch Set 2:

Also would be good to see the Analysis Class work more statefully as current implementation looks to require lots of calls from external tester.  (i.e. do cbtransfer, compare maps…)  Where if the Class could make calls to itself then the test could do:

analyzer = DataResultAnylizer(hostname, user, password)
analyzer.report()
  # goes and does cbtransfer and comparisons…etc and reports

-------------------------------------
author: Quentin Ali
date: 2014-04-08 19:40:05.957000000

Patch Set 2:

Thanks, Tommie, working on it

-------------------------------------
author: Linda Vaughan
date: 2014-04-08 20:10:31.847000000

Patch Set 2: Code-Review-1

(3 comments)

Line:1947, lib/remote/remote_util.py -> The 'open' looks redundant the file was already opened above and we can do this operations in 'with' context?

Line:1973, lib/remote/remote_util.py -> The 'for' loop is common between the two functions-cbtransfer_disk and cbtransfer_memory, can you make this into one function.

Line:5, lib/couchbase/data_analysis_helper.py -> Or if these variables will only be used in Object then make them instance variables.

-------------------------------------
author: Quentin Ali
date: 2014-04-08 22:36:31.855000000

Patch Set 3: Patch Set 2 was rebased

-------------------------------------
