DESCRIPTION

CBQE-0: fix failover test cases due with data dump comparison being set to disk+memory

Change-Id: I6ddf75d36dd4511a15633cb880edf1dc51fb0ced


COMMENTS

author: Quentin Ali
date: 2014-05-04 01:37:35.117000000

Uploaded patch set 6.

-------------------------------------
author: Quentin Ali
date: 2014-05-04 01:39:14.971000000

Patch Set 6:

- Fix for graceful failover as default
- Fix for operations during rebalance after failover
- Fix for data analysis during failover if required
- Fix for avoiding stats check for versions < 3.0.0 (applicable only to failover test cases)
- Fix for failure in checking stats in RebalanceHelper

Examples of run

Rebalance In:: https://friendpaste.com/3TmqoNjEF3xnYtbLyau3HA
Rebalance Out:: https://friendpaste.com/3TmqoNjEF3xnYtbLyb2Vr7
RebalanceInOut:: https://friendpaste.com/3TmqoNjEF3xnYtbLyb2Vr7
Failover with Firewall Failure:: https://friendpaste.com/3TmqoNjEF3xnYtbLybBED7
Failover with Firewall::https://friendpaste.com/3TmqoNjEF3xnYtbLyb2ULA

-------------------------------------
author: Hugo Blankenship
date: 2014-05-04 02:00:47.658000000

Patch Set 6:

Build Started http://factory.couchbase.com/job/testrunner-Jasmin Rangel-master/1257/

-------------------------------------
author: Hugo Blankenship
date: 2014-05-04 02:53:23.314000000

Patch Set 6: Verified-1

Build Failed 

http://factory.couchbase.com/job/testrunner-Jasmin Rangel-master/1257/ : ABORTED

detect lines like:

suite viewmergetests.ViewMergingTests , pass 38 , fail 1

-------------------------------------
author: Esperanza Cherry
date: 2014-05-05 07:14:58.878000000

Patch Set 6:

(1 comment)

it's very difficult to read failover's code. with this we need to do something. already at the stage of debug, we have a lot of fixes. 

about test results:
Failover with Firewall Failure:: https://friendpaste.com/3TmqoNjEF3xnYtbLybBED7 Failover with Firewall::https://friendpaste.com/3TmqoNjEF3xnYtbLyb2ULA

these tests runs with dgm=True, but I don't see that we reach dgm in the tests. we check only initial items number(10k).

Line:121, pytests/failover/failovertests.py -> what if not self.ops_tasks? where we wait that these tasks finished? maybe it should be after rebalance completed?

-------------------------------------
author: Quentin Ali
date: 2014-05-05 15:48:05.446000000

Patch Set 6:

Andrei

I don't agree with your 'code too long' comments. The reason is that the failover previously, it was too long and not refactored. It also had missing data analysis missing. Maintaining one body function for everything makes it confusing. Also with incoming scenarios and support for views, data ops it was important that we divided the code. So re-factor is the way to go.

Secondly, I have not designed the DGM Scenario, so can't comment much on it - do you know who designed it?

The current patch, fixes issues with replication achieved, disk queue flush and data analysis for failure scenario. And this is the requirement for my tests. And this requires immediate attention.

My intention is to fix things and get things going. Please understand that Luca Rhodes is currently blocked and the process was started to fix the things mentioned above. 

I would appreciate if the review is not extended beyond two days.

-------------------------------------
author: Esperanza Cherry
date: 2014-05-05 17:20:20.399000000

Patch Set 6:

1)okay, lets check changes on real cluster with real data
2) dgm param works enough ugly now: we create bucket with ram_quota=256 and store N items. in you tests you use 10K and we don't reach dgm. there is another option active_resident_threshold that is 'smarter", but let's wait a little with these shortcomings for failover tests.

"I would appreciate if the review is not extended beyond two days."
I believe that I have never delayed checking commits(if I'm online). 
but if you think that the progress is not so fast, then there are 2 options:
a) commits should be smaller
b) add more reviewers

-------------------------------------
author: Quentin Ali
date: 2014-05-05 17:49:08.071000000

Patch Set 6:

Added more reviewers

-------------------------------------
author: Quentin Ali
date: 2014-05-05 17:58:16.306000000

Patch Set 6:

I will check for DGM per suggestion. IMHO, not a topic for this review. But definitely worth a check and fixed in the failover tests config or in code if required

-------------------------------------
