DESCRIPTION

CBQE-62: improved key/value result verification

adds detection for unexpected and duplicate keys
along with missing keys.  If a key is missing,
memcached is checked and couchdb to ensure
missing key is actually caused by view merge
issue.

Change-Id: Id9673745d7e542f7fd42c807d4699b0c62d8f8d3


COMMENTS

author: Titus Ho
date: 2012-05-23 00:52:51.388000000

Uploaded patch set 3.

-------------------------------------
author: Hugo Blankenship
date: 2012-05-23 00:54:51.248000000

Patch Set 3:

Build Started http://qa.hq.northscale.net/job/testrunner-Jasmin Rangel-master/384/ 

-------------------------------------
author: Hugo Blankenship
date: 2012-05-23 00:59:52.274000000

Patch Set 3: Verified

Build Successful 

http://qa.hq.northscale.net/job/testrunner-Jasmin Rangel-master/384/ : SUCCESS

-------------------------------------
author: Ayla Reynolds
date: 2012-05-23 11:40:19.783000000

Patch Set 3: Looks good to me, approved



-------------------------------------
author: Porter Oconnor
date: 2012-05-23 12:31:04.456000000

Patch Set 3:

Tommie, I can't express how happy I'm seeing this moving forward :)
Thanks a lot for this.

I was just testing this, and it seems it's massively slowing down the tests.
I ran:

$ ./testrunner -i b/resources/dev-4-nodes.ini -c conf/py-viewquery.conf

And the test viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries was stuck during (or after, hard to tell for me) the phase where it creates direct clients for several minutes. See the output at:

https://friendpaste.com/21v0JQbRqFvldikarBgvnR

I had to kill the python process, as it was stuck in that part for more than 10 minutes. Without this change, this slow down doesn't happen.

Do you have any clues about this, or this might be something specific to my environment? (cluster_run, Ubuntu 32 bits)

-------------------------------------
author: Ayla Reynolds
date: 2012-05-23 18:14:14.120000000

Patch Set 3: I would prefer that you didn't submit this

Sorry, I have to change my vote.

The code itself looks great. But the problem we have to deal with now is, that the verification takes a long time.

I think we have a problem that there are two different types of verification. One which means "verify all the data, to make sure it is correct". Sometimes you don't need it, as you load the data in.

And there is the verification if something went wrong. There you should always give a more detailed output, to make debugging easier (this is actually what Filipe originally requested).

Hence my proposal. Keep the default verification to a minimum (e.g. just take the count) and only do deep introspection/validation if there was an error.

-------------------------------------
author: Ayla Reynolds
date: 2012-05-23 18:19:01.002000000

Patch Set 3: (1 inline comment)



-------------------------------------
author: Esperanza Cherry
date: 2012-05-23 19:09:50.539000000

Patch Set 3: (1 inline comment)



Line:1009, pytests/viewquerytests.py -> here should be added verification that determines whether there are duplicates:
if len(ids)!=len(couch_set)

-------------------------------------
author: Titus Ho
date: 2012-05-23 20:06:38.376000000

Patch Set 3:

great feedback everyone, thanks for trying out the code.  

One thing about verification is that it's easy to detect a case when we have duplicate documents, but it takes more work to report exactly which docs are dupes - hence the extra count() which seems to be causing slowdown.  Will be working on performance fixes for next patch.

Also, Filipe in your screenshot looks like you could be hitting problem in data loading where the test tries to build the expected key set from query params so that the verification here can work.  That code was in the base diff.   lots of math and looping there :)

-------------------------------------
author: Tate Garrett
date: 2012-05-30 20:39:50.243000000

Your change could not be merged due to a path conflict.

Please merge (or rebase) the change locally and upload the resolution for review.

-------------------------------------
author: Ayla Reynolds
date: 2012-05-23 18:19:00.996000000

MISMATCHED INLINE COMMENT
Line:1009, pytests/viewquerytests.py -> I identified this one as a huge bottleneck (with putting out some log messages during a testrunner run).

-------------------------------------
