DESCRIPTION

CBQE-545: Conflict resolution (biXDCR)

- Multiple updates from source and
destination on an overlapping set.

Change-Id: I4269d9b026c169373c25755fe8a4b424576370f8


COMMENTS

author: Emerson Nolan
date: 2013-03-04 20:48:36.598000000

Uploaded patch set 2.

-------------------------------------
author: Hugo Blankenship
date: 2013-03-04 21:01:57.759000000

Patch Set 2:

Build Started http://qa.hq.northscale.net/job/testrunner-Jasmin Rangel-master/5521/ 

-------------------------------------
author: Amani Conrad
date: 2013-03-04 21:18:27.571000000

Patch Set 2:

ion(Exception('Bad hash result: %d != %d' % (crc32.crc32_hash(d), int(value))))"), ('lib/tasks/future.py', 263, 'set_exception', 'print traceback.extract_stack()')]
Mon Mar  4 13:08:38 2013
[('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 503, '__bootstrap', 'self.__bootstrap_inner()'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 530, '__bootstrap_inner', 'self.run()'), ('lib/tasks/task.py', 413, 'run', 'self.next()'), ('lib/tasks/task.py', 850, 'next', 'self._check_valid_keys(keys_batch)'), ('lib/tasks/task.py', 869, '_check_valid_keys', 'self._check_validity(partition, keys, key_vals)'), ('lib/tasks/task.py', 884, '_check_validity', "self.set_exception(Exception('Bad hash result: %d != %d' % (crc32.crc32_hash(d), int(value))))"), ('lib/tasks/future.py', 263, 'set_exception', 'print traceback.extract_stack()')]
Mon Mar  4 13:08:38 2013
[('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 503, '__bootstrap', 'self.__bootstrap_inner()'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 530, '__bootstrap_inner', 'self.run()'), ('lib/tasks/task.py', 413, 'run', 'self.next()'), ('lib/tasks/task.py', 850, 'next', 'self._check_valid_keys(keys_batch)'), ('lib/tasks/task.py', 869, '_check_valid_keys', 'self._check_validity(partition, keys, key_vals)'), ('lib/tasks/task.py', 884, '_check_validity', "self.set_exception(Exception('Bad hash result: %d != %d' % (crc32.crc32_hash(d), int(value))))"), ('lib/tasks/future.py', 263, 'set_exception', 'print traceback.extract_stack()')]
Mon Mar  4 13:08:38 2013
[2013-03-04 13:08:38,861] - [rest_client:657] INFO - removing remote cluster name:cluster1
[('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 503, '__bootstrap', 'self.__bootstrap_inner()'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 530, '__bootstrap_inner', 'self.run()'), ('lib/tasks/task.py', 413, 'run', 'self.next()'), ('lib/tasks/task.py', 850, 'next', 'self._check_valid_keys(keys_batch)'), ('lib/tasks/task.py', 869, '_check_valid_keys', 'self._check_validity(partition, keys, key_vals)'), ('lib/tasks/task.py', 884, '_check_validity', "self.set_exception(Exception('Bad hash result: %d != %d' % (crc32.crc32_hash(d), int(value))))"), ('lib/tasks/future.py', 263, 'set_exception', 'print traceback.extract_stack()')]
Mon Mar  4 13:08:38 2013
[('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 503, '__bootstrap', 'self.__bootstrap_inner()'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 530, '__bootstrap_inner', 'self.run()'), ('lib/tasks/task.py', 413, 'run', 'self.next()'), ('lib/tasks/task.py', 850, 'next', 'self._check_valid_keys(keys_batch)'), ('lib/tasks/task.py', 869, '_check_valid_keys', 'self._check_validity(partition, keys, key_vals)'), ('lib/tasks/task.py', 884, '_check_validity', "self.set_exception(Exception('Bad hash result: %d != %d' % (crc32.crc32_hash(d), int(value))))"), ('lib/tasks/future.py', 263, 'set_exception', 'print traceback.extract_stack()')]
Mon Mar  4 13:08:38 2013
[('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 503, '__bootstrap', 'self.__bootstrap_inner()'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 530, '__bootstrap_inner', 'self.run()'), ('lib/tasks/task.py', 413, 'run', 'self.next()'), ('lib/tasks/task.py', 850, 'next', 'self._check_valid_keys(keys_batch)'), ('lib/tasks/task.py', 869, '_check_valid_keys', 'self._check_validity(partition, keys, key_vals)'), ('lib/tasks/task.py', 884, '_check_validity', "self.set_exception(Exception('Bad hash result: %d != %d' % (crc32.crc32_hash(d), int(value))))"), ('lib/tasks/future.py', 263, 'set_exception', 'print traceback.extract_stack()')]
Mon Mar  4 13:08:38 2013
[('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 503, '__bootstrap', 'self.__bootstrap_inner()'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 530, '__bootstrap_inner', 'self.run()'), ('lib/tasks/task.py', 413, 'run', 'self.next()'), ('lib/tasks/task.py', 850, 'next', 'self._check_valid_keys(keys_batch)'), ('lib/tasks/task.py', 869, '_check_valid_keys', 'self._check_validity(partition, keys, key_vals)'), ('lib/tasks/task.py', 884, '_check_validity', "self.set_exception(Exception('Bad hash result: %d != %d' % (crc32.crc32_hash(d), int(value))))"), ('lib/tasks/future.py', 263, 'set_exception', 'print traceback.extract_stack()')]
Mon Mar  4 13:08:38 2013
[2013-03-04 13:08:38,928] - [rest_client:657] INFO - removing remote cluster name:cluster0
[2013-03-04 13:08:39,004] - [xdcrbasetests:396] INFO - cleanup cluster1: [ip:127.0.0.1 port:9000 ssh_username:Administrator, ip:127.0.0.1 port:9001 ssh_username:Administrator]
[2013-03-04 13:08:39,024] - [bucket_helper:136] INFO - deleting existing buckets [u'default'] on 127.0.0.1
[2013-03-04 13:08:39,025] - [bucket_helper:138] INFO - remove bucket default ...
[2013-03-04 13:08:39,382] - [bucket_helper:146] INFO - deleted bucket : default from 127.0.0.1
[2013-03-04 13:08:39,383] - [bucket_helper:216] INFO - waiting for bucket deletion to complete....
[2013-03-04 13:08:39,386] - [rest_client:109] INFO - existing buckets : []
[2013-03-04 13:08:39,398] - [cluster_helper:259] INFO - rebalancing all nodes in order to remove nodes
[2013-03-04 13:08:39,401] - [rest_client:804] INFO - rebalance params : password=asdasd&ejectedNodes=n_1%4010.17.39.63&user=Administrator&knownNodes=n_1%4010.17.39.63%2Cn_0%4010.17.39.63
[2013-03-04 13:08:39,404] - [rest_client:808] INFO - rebalance operation started
[2013-03-04 13:08:39,408] - [rest_client:905] INFO - rebalance percentage : 0 %
[2013-03-04 13:08:43,416] - [rest_client:864] INFO - rebalance progress took 4.01105880737 seconds 
[2013-03-04 13:08:43,416] - [rest_client:865] INFO - sleep for 4.01105880737 seconds after rebalance...
[2013-03-04 13:08:47,439] - [cluster_helper:288] INFO - removed all the nodes from cluster associated with ip:127.0.0.1 port:9000 ssh_username:Administrator ? xxx@xxx.xxx 9001)]
[2013-03-04 13:08:47,443] - [cluster_helper:84] INFO - waiting for ns_server @ 127.0.0.1:9000
[2013-03-04 13:08:47,448] - [cluster_helper:86] INFO - ns_server @ 127.0.0.1:9000 is running
[2013-03-04 13:08:47,453] - [bucket_helper:136] INFO - deleting existing buckets [] on 127.0.0.1
[2013-03-04 13:08:47,465] - [cluster_helper:84] INFO - waiting for ns_server @ 127.0.0.1:9001
[2013-03-04 13:08:47,468] - [cluster_helper:86] INFO - ns_server @ 127.0.0.1:9001 is running
[2013-03-04 13:08:47,468] - [xdcrbasetests:396] INFO - cleanup cluster2: [ip:127.0.0.1 port:9002 ssh_username:Administrator, ip:127.0.0.1 port:9003 ssh_username:Administrator]
[2013-03-04 13:08:47,477] - [bucket_helper:136] INFO - deleting existing buckets [u'default'] on 127.0.0.1
[2013-03-04 13:08:47,477] - [bucket_helper:138] INFO - remove bucket default ...
[2013-03-04 13:08:47,832] - [bucket_helper:146] INFO - deleted bucket : default from 127.0.0.1
[2013-03-04 13:08:47,832] - [bucket_helper:216] INFO - waiting for bucket deletion to complete....
[2013-03-04 13:08:47,835] - [rest_client:109] INFO - existing buckets : []
[2013-03-04 13:08:47,847] - [cluster_helper:259] INFO - rebalancing all nodes in order to remove nodes
[2013-03-04 13:08:47,850] - [rest_client:804] INFO - rebalance params : password=asdasd&ejectedNodes=n_3%4010.17.39.63&user=Administrator&knownNodes=n_3%4010.17.39.63%2Cn_2%4010.17.39.63
[2013-03-04 13:08:47,853] - [rest_client:808] INFO - rebalance operation started
[2013-03-04 13:08:47,855] - [rest_client:905] INFO - rebalance percentage : 0 %
[2013-03-04 13:08:51,862] - [rest_client:864] INFO - rebalance progress took 4.00872802734 seconds 
[2013-03-04 13:08:51,862] - [rest_client:865] INFO - sleep for 4.00872802734 seconds after rebalance...
[2013-03-04 13:08:55,884] - [cluster_helper:288] INFO - removed all the nodes from cluster associated with ip:127.0.0.1 port:9002 ssh_username:Administrator ? xxx@xxx.xxx 9003)]
[2013-03-04 13:08:55,889] - [cluster_helper:84] INFO - waiting for ns_server @ 127.0.0.1:9002
[2013-03-04 13:08:55,893] - [cluster_helper:86] INFO - ns_server @ 127.0.0.1:9002 is running
[2013-03-04 13:08:55,898] - [bucket_helper:136] INFO - deleting existing buckets [] on 127.0.0.1
[2013-03-04 13:08:55,910] - [cluster_helper:84] INFO - waiting for ns_server @ 127.0.0.1:9003
[2013-03-04 13:08:55,913] - [cluster_helper:86] INFO - ns_server @ 127.0.0.1:9003 is running
[2013-03-04 13:08:55,913] - [xdcrbasetests:145] INFO - ==============  XDCRbasetests cleanup was finished for test #3 load_with_async_ops_and_joint_sets ==============

======================================================================
ERROR: load_with_async_ops_and_joint_sets (xdcr.biXDCR.bidirectional)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/xdcr/biXDCR.py", line 87, in load_with_async_ops_and_joint_sets
    self.verify_results(verify_src=True)
  File "pytests/xdcr/xdcrbasetests.py", line 659, in verify_results
    self.verify_xdcr_stats(src_nodes, dest_nodes, verify_src)
  File "pytests/xdcr/xdcrbasetests.py", line 638, in verify_xdcr_stats
    self._verify_all_buckets(self.src_master)
  File "pytests/xdcr/xdcrbasetests.py", line 1019, in _verify_all_buckets
    is_verified = self._poll_for_condition(verify)
  File "pytests/xdcr/xdcrbasetests.py", line 484, in _poll_for_condition
    return self._poll_for_condition_rec(condition, interval, num_itr)
  File "pytests/xdcr/xdcrbasetests.py", line 490, in _poll_for_condition_rec
    if condition():
  File "pytests/xdcr/xdcrbasetests.py", line 1012, in verify
    task.result(timeout)
  File "lib/tasks/future.py", line 159, in result
    return self.__get_result()
  File "lib/tasks/future.py", line 111, in __get_result
    raise self._exception
Exception: Bad hash result: 31540 != 15063

----------------------------------------------------------------------
Ran 1 test in 324.944s


Can you check the test code on this?

The input "doc-ops=create-update,doc-ops-dest=create-update" will update the same key once on each source and destination cluster. This means, the conflicts will be resolved on basis of CAS and not sequence id.

The above hash result mismatch could be a race condition. Can you confirm?

-------------------------------------
author: Amani Conrad
date: 2013-03-04 21:18:33.643000000

Patch Set 2: I would prefer that you didn't submit this



-------------------------------------
author: Hugo Blankenship
date: 2013-03-04 21:38:58.780000000

Patch Set 2:

Build Failed 

http://qa.hq.northscale.net/job/testrunner-Jasmin Rangel-master/5521/ : ABORTED

-------------------------------------
