DESCRIPTION

CBQE:0 Fix not printing of missing keys + add pre-graceful failover active-replica key count

Change-Id: I526e4c8da76148f7d2f39ed7b09296c5a1b873c4


COMMENTS

author: Hugo Blankenship
date: 2014-11-25 00:38:35.171000000

Patch Set 2:

Build Started http://factory.couchbase.com/job/testrunner-Jasmin Rangel-master/3147/

-------------------------------------
author: Hugo Blankenship
date: 2014-11-25 00:54:34.268000000

Patch Set 2: Verified+1

Build Successful 

http://factory.couchbase.com/job/testrunner-Jasmin Rangel-master/3147/ : SUCCESS

-------------------------------------
author: Fisher Vargas
date: 2014-11-25 08:32:33.832000000

Patch Set 2: Code-Review-1

(3 comments)

Line:633, pytests/xdcr/xdcrbasetests.py -> It is very huge value to print.

Line:686, pytests/xdcr/xdcrbasetests.py -> Its was synchronous failover before. but it is asynronous failover?

Line:1331, pytests/xdcr/xdcrbasetests.py -> Please add argument for the function, skip_timeout_error=True/False rather.

-------------------------------------
author: Angelina Mayer
date: 2014-11-26 17:59:34.112000000

Patch Set 2:

(3 comments)

Line:633, pytests/xdcr/xdcrbasetests.py -> we are only printing the number of keys loaded, not the list.

Line:686, pytests/xdcr/xdcrbasetests.py -> instead of calling the cluster method directly, I'm having it call a base class class method. Async is any day better than sync right

Line:1331, pytests/xdcr/xdcrbasetests.py -> will there ever be a case where we need to break here and not print the missing keys?

-------------------------------------
author: Fisher Vargas
date: 2014-11-27 10:39:14.851000000

Patch Set 2:

(2 comments)

Line:686, pytests/xdcr/xdcrbasetests.py -> We can not perform failover and rebalance in parallel for the same node, first failover should be done and then rebalance.

Line:1331, pytests/xdcr/xdcrbasetests.py -> No, but here Timeout error caught and will not throw. Either use "raise" statement after self.log.info to re-throw the exception.

-------------------------------------
author: Angelina Mayer
date: 2014-12-01 21:09:33.981000000

Patch Set 2:

(2 comments)

Line:686, pytests/xdcr/xdcrbasetests.py -> Looks like there is no "synchronous" failover. Even cluster.failover internally calls async_failover. So keeping _async_failover for added logging. However completing failover before rebalance (in next patch).

2014-12-01 13:01:53 | INFO | MainProcess | test_thread | [xdcrbasetests._async_failover] Printing pre-failover stats
2014-12-01 13:01:53 | INFO | MainProcess | test_thread | [xdcrbasetests._get_active_replica_count_from_cluster] Keys loaded into bucket default:7744
2014-12-01 13:01:53 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2014-12-01 13:01:53 | INFO | MainProcess | test_thread | [xdcrbasetests._get_active_replica_count_from_cluster] Cluster stat: vb_active_curr_items = 7594
2014-12-01 13:01:53 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2014-12-01 13:01:53 | INFO | MainProcess | test_thread | [xdcrbasetests._get_active_replica_count_from_cluster] Cluster stat: vb_replica_curr_items = 7732
2014-12-01 13:01:53 | INFO | MainProcess | test_thread | [xdcrbasetests._async_failover] Gracefully failing over node(s) [ip:127.0.0.1 port:9001 ssh_username:Administrator]
2014-12-01 13:01:54 | INFO | MainProcess | Cluster_Thread | [task._failover_nodes] Failing over 127.0.0.1:9001 with graceful=True
2014-12-01 13:01:54 | INFO | MainProcess | Cluster_Thread | [rest_client.fail_over] fail_over node xxx@xxx.xxx successful
2014-12-01 13:01:54 | INFO | MainProcess | Cluster_Thread | [task.execute] 0 seconds sleep after failover, for nodes to go pending....
2014-12-01 13:01:54 | INFO | MainProcess | test_thread | [xdcrbasetests.__async_rebalance_out_cluster]  Starting rebalance-out nodes:['127.0.0.1'] at source cluster 127.0.0.1
2014-12-01 13:01:54 | INFO | MainProcess | test_thread | [xdcrbasetests._async_failover] Printing pre-failover stats
2014-12-01 13:01:54 | INFO | MainProcess | test_thread | [xdcrbasetests._get_active_replica_count_from_cluster] Keys loaded into bucket default:8329
2014-12-01 13:01:54 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12004 default
2014-12-01 13:01:54 | INFO | MainProcess | test_thread | [xdcrbasetests._get_active_replica_count_from_cluster] Cluster stat: vb_active_curr_items = 7914
2014-12-01 13:01:54 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12004 default
2014-12-01 13:01:55 | INFO | MainProcess | test_thread | [xdcrbasetests._get_active_replica_count_from_cluster] Cluster stat: vb_replica_curr_items = 7981
2014-12-01 13:01:55 | INFO | MainProcess | test_thread | [xdcrbasetests._async_failover] Gracefully failing over node(s) [ip:127.0.0.1 port:9004 ssh_username:Administrator]
2014-12-01 13:01:55 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=welcome&ejectedNodes=n_1%40127.0.0.1&user=Administrator&knownNodes=n_0%4010.17.2.125%2Cn_1%40127.0.0.1
2014-12-01 13:01:55 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
2014-12-01 13:01:55 | INFO | MainProcess | Cluster_Thread | [task._failover_nodes] Failing over 127.0.0.1:9004 with graceful=True
2014-12-01 13:01:55 | INFO | MainProcess | Cluster_Thread | [rest_client.fail_over] fail_over node xxx@xxx.xxx successful
2014-12-01 13:01:55 | INFO | MainProcess | Cluster_Thread | [task.execute] 0 seconds sleep after failover, for nodes to go pending....
2014-12-01 13:01:55 | INFO | MainProcess | test_thread | [xdcrbasetests.__async_rebalance_out_cluster]  Starting rebalance-out nodes:['127.0.0.1'] at source cluster 127.0.0.1
2014-12-01 13:01:55 | INFO | MainProcess | test_thread | [xdcrbasetests.sleep] sleep for 5 secs.  ...
2014-12-01 13:01:55 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 84.38 %
2014-12-01 13:01:55 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=welcome&ejectedNodes=n_4%40127.0.0.1&user=Administrator&knownNodes=n_2%4010.17.2.125%2Cn_4%40127.0.0.1
2014-12-01 13:01:55 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
2014-12-01 13:01:55 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %

Line:1331, pytests/xdcr/xdcrbasetests.py -> Thats the whole point. If we raise an exception here, the metadata validation does not happen. We don't want to call timeout an error. That doesn't provide useful information. We want to point to the resulting metadata mismatch as error.

-------------------------------------
author: Fisher Vargas
date: 2014-12-02 06:16:47.433000000

Patch Set 2:

(2 comments)

Line:686, pytests/xdcr/xdcrbasetests.py -> mmm
    def failover(self, servers=[], failover_nodes=[], graceful=False, use_hostnames=False):
        """Synchronously flushes a bucket

        Parameters:
            servers - node used for connection (TestInputServer)
            failover_nodes - servers to be failover. (TestInputServer)
            bucket - The name of the bucket to be flushed. (String)

        Returns:
            boolean - Whether or not the bucket was flushed."""
        _task = self.async_failover(servers, failover_nodes, graceful, use_hostnames)
        return _task.result()

Please check _task.result() is waiting for failover to finish here.

Line:1331, pytests/xdcr/xdcrbasetests.py -> I believe metadata verification is written under "finally" clause. Ideally it execute finally statement, but its not happening it should be different issue.

-------------------------------------
