DESCRIPTION

CBQE-0: enhance failover and rebalance logic with fixes to data_analysis logic

Change-Id: I39e8b65521414b872ded92b71ebe8f93a3b25199


COMMENTS

author: Hugo Blankenship
date: 2014-04-26 02:19:00.938000000

Patch Set 13:

Build Started http://factory.couchbase.com/job/testrunner-Jasmin Rangel-master/1165/

-------------------------------------
author: Esperanza Cherry
date: 2014-04-26 02:19:02.681000000

Patch Set 13:

Automatically re-added by Roselyn Villegas trivial rebase detection script.

-------------------------------------
author: Quentin Ali
date: 2014-04-26 02:20:35.203000000

Patch Set 13:

Addressed Andrei’s comments

For the following question

Line 386:     def create_file(self,chosen,buckets,serverMap):

what is the purpose to create these files?

The check file is added to compare delta/full rebalance to see if a data folder is actually recreated. The paramiko client was not giving a consistent result using modified time. So had to go via this route.


Examples of Test Runs
_______________________

1) Failover Tests


Delta Recovery with Graceful failover

./testrunner -i ~/ini/palm.ini -t failover.failovertests.FailoverTests.test_failover_then_add_back,replicas=2,num_failed_nodes=1,items=10000,vbuckets=64,graceful=True,max_verify=10,upr=True,recoveryType=delta,graceful=True,withOps=True

https://friendpaste.com/976HJat3vsgve0krGoMVo 

Views Use Example

 ./testrunner -i ~/ini/palm.ini -t failover.failovertests.FailoverTests.test_failover_stop_server,replicas=1,num_failed_nodes=1,upr=True,doc_ops=update:create:delete,withQueries=True,numViews=5,runViews=True

https://friendpaste.com/976HJat3vsgve0krGXMGv

Master Failover Example

./testrunner -i ~/ini/palm.ini -t  failover.failovertests.FailoverTests.test_failover_then_add_back,replicas=2,num_failed_nodes=1,items=10000,vbuckets=64,graceful=True,max_verify=10,upr=True,recoveryType=full,graceful=True,failoverMaster=True,withOps=True

https://friendpaste.com/976HJat3vsgve0krGXL9S

Firewall Example (this failing as of now and is an open bug)

https://www.couchbase.com/issues/browse/MB-10934

 ./testrunner -i ~/ini/palm.ini -t failover.failovertests.FailoverTests.test_failover_firewall,replicas=1,vbuckets=128,num_failed_nodes=1,items=1000,upr=True

https://friendpaste.com/976HJat3vsgve0krGXJzD

2) Rebalance-In Tests

No extra operations except data loading

./testrunner -i ~/ini/palm.ini -t rebalance.rebalancein.RebalanceInTests.rebalance_in_after_ops,nodes_in=1,nodes_init=3,replicas=1,items=1000,vbuckets=64,upr=True

https://friendpaste.com/x6fj3qswF88iKCMTrAlo 

3) Rebalance-Out Tests

No extra operations except data loading

./testrunner -i ~/ini/palm.ini -t rebalance.rebalanceout.RebalanceOutTests.rebalance_out_after_ops,nodes_out=1,replicas=1,items=10000,vbuckets=64,upr=True

https://friendpaste.com/976HJat3vsgve0krGXO4g

4) Rebalance-in-out Tests

No extra operations except data loading

./testrunner -i ~/ini/palm.ini -t rebalance.rebalanceinout.RebalanceInOutTests.rebalance_in_out_after_mutation,nodes_in=1,nodes_init=3,replicas=1,items=100000,vbuckets=64,upr=True

https://friendpaste.com/976HJat3vsgve0krGXIpd

-------------------------------------
author: Hugo Blankenship
date: 2014-04-26 02:44:13.931000000

Patch Set 13: Verified-1

Build Failed 

http://factory.couchbase.com/job/testrunner-Jasmin Rangel-master/1165/ : FAILURE

-------------------------------------
author: Esperanza Cherry
date: 2014-04-27 08:50:27.439000000

Patch Set 13:

"The paramiko client was not giving a consistent result using modified time."

are u tNathalie Landrying about paramiko_client.open_sftp().stat(PATH_TO_FOLDER).st_mtime ?

-------------------------------------
author: Quentin Ali
date: 2014-04-27 18:05:55.181000000

Patch Set 13:

yes, m_time changes for data dir and couch files: for both delta vs full recovery

-------------------------------------
author: Quentin Ali
date: 2014-04-27 20:39:00.383000000

Patch Set 14: Patch Set 13 was rebased

-------------------------------------
