DESCRIPTION

CBQE-2182: added steps to collect data files and run RevId Verification
for Xdcr tests

Change-Id: Ic456b03ed5013c8d30a6114663f7ddf021d9e533
Reviewed-on: http://review.couchbase.org/35670
Reviewed-by: Esperanza Cherry xxx@xxx.xxx
Reviewed-by: Fisher Vargas <Fisher xxx@xxx.xxx
Tested-by: Fisher Vargas <Fisher xxx@xxx.xxx


COMMENTS

author: Fisher Vargas
date: 2014-04-18 09:14:43.562000000

Change has been successfully cherry-picked as e9c376fe4bf1de810dd5a16368eee502173c382a

-------------------------------------
author: Angelina Mayer
date: 2014-04-23 00:22:03.475000000

Patch Set 16:

(1 comment)

Line:1401, pytests/xdcr/xdcrbasetests.py -> One thing I don't understand is - self.collect_data_files is set to True and False in the beginning and end of this method unconditionally. But teardown only happens after this method is executed not between setting the variable to true and false. This results in data files never being collected even in mismatch scenarios.

2014-04-22 11:04:42 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne9389 =====
2014-04-22 11:04:42 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:3, Error Count:7
2014-04-22 11:04:42 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:1535964422474215, Destination cas:1535964422474216, Error Count:8
2014-04-22 11:04:42 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data: {'deleted': 1, 'seqno': 2, 'cas': 1535964422474215, 'flags': 0, 'expiration': 1398188779}
2014-04-22 11:04:42 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest  meta  data: {'deleted': 1, 'seqno': 3, 'cas': 1535964422474216, 'flags': 0, 'expiration': 1398188731}
2014-04-22 11:04:42 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 13999 == 14000 expected on '172.23.106.45:8091''172.23.106.46:8091', default bucket
2014-04-22 11:04:42 | INFO | MainProcess | load_gen_task | [task.has_next] Verification done, 17000 items have been verified (3000 of them deleted)
FAIL
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [rebalanceXDCR.tearDown] ==============  XDCRbasetests stats for test #1 async_rebalance_out ==============
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [rebalanceXDCR.tearDown] Type of run: BIDIRECTIONAL XDCR
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] STATS with source at 172.23.106.45 and destination at 172.23.106.47
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] Bucket: default
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Average local replica creation rate for bucket 'default': 33.1874371234 KB per second
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Median XDC replication ops for bucket 'default': 0.631 K ops per second
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Mean XDC replication ops for bucket 'default': 0.686947115581 K ops per second
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Average XDCR data replication rate for bucket 'default': 32.8257327327 KB per second
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] STATS with source at 172.23.106.47 and destination at 172.23.106.45
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] Bucket: default
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Average local replica creation rate for bucket 'default': 31.2656969766 KB per second
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Median XDC replication ops for bucket 'default': 0.0885311871227 K ops per second
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Mean XDC replication ops for bucket 'default': 0.24540443094 K ops per second
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [xdcrbasetests._print_stats] 	Average XDCR data replication rate for bucket 'default': 31.2116690665 KB per second
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [rebalanceXDCR.tearDown] ============== = = = = = = = = END = = = = = = = = = = ==============
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [rebalanceXDCR.tearDown] ==============  rebalanceXDCR cleanup was started for test #1 async_rebalance_out ==============
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [u'default'] on 172.23.106.45
2014-04-22 11:04:42 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] remove bucket default ...
2014-04-22 11:04:46 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.106.45
2014-04-22 11:04:46 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2014-04-22 11:04:46 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] existing buckets : []
2014-04-22 11:04:46 | INFO | MainProcess | MainThread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes
2014-04-22 11:04:46 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=ns_1%40172.23.106.46&user=Administrator&knownNodes=ns_1%40172.23.106.46%2Cns_1%40172.23.106.45
2014-04-22 11:04:46 | INFO | MainProcess | MainThread | [rest_client.rebalance] rebalance operation started

-------------------------------------
author: Fisher Vargas
date: 2014-04-23 06:21:23.338000000

Patch Set 16:

(1 comment)

Line:1401, pytests/xdcr/xdcrbasetests.py -> Do you have complete test logs? Please give me the link after copying friendpaste.com?

-------------------------------------
author: Fisher Vargas
date: 2014-04-23 06:26:13.334000000

Patch Set 16:

(1 comment)

Line:1401, pytests/xdcr/xdcrbasetests.py -> Used simple logic: 

1. It is set to False in the end of the function, If control reaches to line: 1401, which means data verification is successful and no exception is throw. and set to False as there is no need collect data files further.


2. It was set to True in the beginning of function, if verification is failed after this statement but before line 1401, then control will jump to tearDown() and data files will be collected.

If I have left any use-case, please do mention here so that I can correct it further.

-------------------------------------
author: Angelina Mayer
date: 2014-04-23 15:03:11.849000000

Patch Set 16:

(1 comment)

Line:1401, pytests/xdcr/xdcrbasetests.py -> Pls look at http://qa.sc.couchbase.com/view/3.0.0/job/centos_x64--00_00-XDCR-UPR-sanity/11/consoleFull

There are 2 testcases that failed due to item mismatch(grep:mismatch). In neither of the two scenarios, data files were collected. In your testing, were you able to collect data files?

-------------------------------------
author: Fisher Vargas
date: 2014-04-23 16:33:59.279000000

Patch Set 16:

(1 comment)

Line:1401, pytests/xdcr/xdcrbasetests.py -> I got it, there is tearDown in rebalanceXDCR.py which doesn't call XDCRBaseTest's tearDown that is why data files were not collected. I will fix it separately, thanks for catching this issue early.

-------------------------------------
author: Angelina Mayer
date: 2014-04-23 17:26:06.287000000

Patch Set 16:

(1 comment)

Line:1401, pytests/xdcr/xdcrbasetests.py -> xdcrbase test teardown is also failing. Pls see XDCR upr sanity job(latest). Can you also enclose the collect data files in try except, add a log stmt that we are collecting data files, so we know why teardown is failing?

Also, we now seem to be collecting data files even for disk drain queue not getting flushed or when other basic calls fail. We can turn on self.collect_data_files = True in except block as in - 
http://review.couchbase.org/#/c/36167/   so we collect only when item counts/values don't match.

-------------------------------------
