DESCRIPTION

CBQE:2261 Unit tests for XDCR checkpointing

These are unit tests for checkpointing that test if vb_uuids are changing with node crash(UPR,TAP),
rebalance(TAP) and failover(UPR,TAP) at source/destination, validate checkpoint records and whether
_pre_replicates and _commit_for_checkpoint calls were successful, whether a checkpoint call
failed/succeeded as expected and if XDCR recovers from expected checkpointing failures.

Today, due to lack of vb-level checkpointing stats or other infrastructure that ns_server has promised in future,
these tests load mutations onto vb0 and test the above.

Change-Id: I3ef5d69f2e98b160afa75e095e484f0fdbdf261a


COMMENTS

author: Angelina Mayer
date: 2014-05-29 18:08:42.719000000

Uploaded patch set 8.

-------------------------------------
author: Hugo Blankenship
date: 2014-05-29 18:08:49.696000000

Patch Set 8:

Build Started http://factory.couchbase.com/job/testrunner-Jasmin Rangel-master/1623/

-------------------------------------
author: Hugo Blankenship
date: 2014-05-29 18:35:30.657000000

Patch Set 8: Verified+1

Build Successful 

http://factory.couchbase.com/job/testrunner-Jasmin Rangel-master/1623/ : SUCCESS

-------------------------------------
author: Fisher Vargas
date: 2014-06-02 17:09:46.689000000

Patch Set 8:

(20 comments)

Most of them are trivial comments. I will review more on this again.

Line:15, pytests/xdcr/checkpointXDCR.py -> Is it possible to order the functions. E.g.

setup()

tearDown()

internal function used by test cases()

verification function()

test cases.....() ?

Line:21, pytests/xdcr/checkpointXDCR.py -> Can we start making function private/public to seggregate their visibility and better readability..

Line:46, pytests/xdcr/checkpointXDCR.py -> these variables are local to this function. no need to use self.xxxx here. only xxx is sufficient. Help to understand data members and local variables.

Line:97, pytests/xdcr/checkpointXDCR.py -> Can add timestamp here, when we have recorded this checkpoint to help dev to investigate.

self.chkpt_records[time.time()] = checkpoint_record

Line:147, pytests/xdcr/checkpointXDCR.py -> Path will not work in Windows.

Line:148, pytests/xdcr/checkpointXDCR.py -> This will search in all the files, we can search in xdcr files as xdcr*.*.

Line:150, pytests/xdcr/checkpointXDCR.py -> Please use log statement instead of print.

Line:152, pytests/xdcr/checkpointXDCR.py -> Please use log instead of print.

Line:235, pytests/xdcr/checkpointXDCR.py -> Is n, num_mutations?

Line:240, pytests/xdcr/checkpointXDCR.py -> this sleep should be in the last of while statement e.g. after count += 1. Is it mutation and wait, right?

Line:244, pytests/xdcr/checkpointXDCR.py -> log statement is already present in was_checkpointing_successful() function.

Line:315, pytests/xdcr/checkpointXDCR.py -> I think, all the tests are based on "default" bucket, can you please ensure that in all tests by overriding _create_buckets function()?

Line:316, pytests/xdcr/checkpointXDCR.py -> it will create buckets, mentioned in the test_params, sasl and standard too.

Line:318, pytests/xdcr/checkpointXDCR.py -> Please use _replication_clusters from xdcrbasetests.py

Line:321, pytests/xdcr/checkpointXDCR.py -> why it is reset to zero here?

Line:341, pytests/xdcr/checkpointXDCR.py -> Will it rebalance-out this node with vb0?

Line:422, pytests/xdcr/checkpointXDCR.py -> _verify_revIds (from xdcrbasetests.py) function could be used here?

Line:430, pytests/xdcr/checkpointXDCR.py -> We can avoid getMeta() call twice here. 1 call is here and second in next if statement.

Line:437, pytests/xdcr/checkpointXDCR.py -> Please log what field is mismtach in RevID here.

Line:442, pytests/xdcr/checkpointXDCR.py -> If print the missing keys also for investigation.

-------------------------------------
author: Esperanza Cherry
date: 2014-06-03 19:39:19.568000000

Patch Set 8:

(3 comments)

Line:1827, lib/membase/api/rest_client.py -> if it's not easy to get replication type we should take any existing bucket from еру cluster. "default" bucket is not always used in the tests

Line:1, pytests/xdcr/checkpointXDCR.py -> never used

Line:315, pytests/xdcr/checkpointXDCR.py -> we shouldn't override _create_buckets. if in current implementation we support only "default" then we should check in presetup that default_bucket=true otherwise fail test. I think we should support all buckets( now or then..) and it's better to pass bucket name in method's params

-------------------------------------
author: Angelina Mayer
date: 2014-06-03 21:59:23.334000000

Patch Set 8:

(20 comments)

Line:1, pytests/xdcr/checkpointXDCR.py -> Done

Line:15, pytests/xdcr/checkpointXDCR.py -> yes, ordered to some extent, not completely though.

Line:21, pytests/xdcr/checkpointXDCR.py -> This being a unit test class with limited abilities, I do not expect this class to be largely inherited by test classes. So it may not make a difference.

Line:46, pytests/xdcr/checkpointXDCR.py -> I intended these to be class variables, one is being used outside this method. Others might be used in future. Added initialization in init()

Line:97, pytests/xdcr/checkpointXDCR.py -> timestamp is a part of record itself. end_time records when exactly checkpointing happened.

Line:147, pytests/xdcr/checkpointXDCR.py -> Done

Line:148, pytests/xdcr/checkpointXDCR.py -> ok

Line:150, pytests/xdcr/checkpointXDCR.py -> ok

Line:152, pytests/xdcr/checkpointXDCR.py -> ok

Line:235, pytests/xdcr/checkpointXDCR.py -> yes, or num of keys

Line:240, pytests/xdcr/checkpointXDCR.py -> no, this method might be called post topology change too in which case, if it is not 60s past the last checkpoint, checkpointing will not happen. Waiting for 60s before inducing mutation can ensure checkpointing will always happen. Also, when replication is first created, a checkpoint for seq 0 is created. If you wait for another 60s to insert a key, another checkpoint is created. So this is intentional.

Line:244, pytests/xdcr/checkpointXDCR.py -> Done

Line:315, pytests/xdcr/checkpointXDCR.py -> Agree with Andrei.

Line:318, pytests/xdcr/checkpointXDCR.py -> I tried using but it results in replication being created twice causing an exception. Didn't dig into that further. Since RestConnection class is anyway used here, using its method doesn't hurt.

Line:321, pytests/xdcr/checkpointXDCR.py -> because the source bucket is being deleted and recreated in which case it will lose its keys. So we reset key_counter = 0 and set keys_loaded to empty list.

Line:341, pytests/xdcr/checkpointXDCR.py -> Ideally if it's a 2 node cluster, and the node containing the vb0 is not the master, then we call self._async_rebalance_out() which will rebalance out non-master node. These are unit tests, I dont intend using more than 2 nodes per cluster for the same.

Line:422, pytests/xdcr/checkpointXDCR.py -> No, we load very few keys(less than 6). In such cases, a simple loop should do instead of incurring the overhead of too many function calls to achieve the same result.

Line:430, pytests/xdcr/checkpointXDCR.py -> ok

Line:437, pytests/xdcr/checkpointXDCR.py -> Again with 5-6 keys, that level of dissection may not be needed. When revID verification fails, just by looking at the metadata once can say what field is incorrect.

Line:442, pytests/xdcr/checkpointXDCR.py -> We print that above.
self.log.error("Key {} is missing at destination".format(key))
This is just to fail the test if there is atleast one missing key.

-------------------------------------
