DESCRIPTION

Refactored KVStore and document generation tasks

This is a rather large change and address a few areas that are
all pretty connected.

KVStore:
This change completly re-writes the kvstore to allow it to run
with multiple threads accessing it at the same time. This is
accomplished with a partition strategy which means that touching
the kvstore only requires locking part of it.

Data loding tasks:
The data loading taks here allow a user to run data loads over
certain key spaces or allow the user to randomly do operations
on key that are contained in the kvstore.

VBucketAwareMemcached:
I also added some functions here that will allow you to do a few
of the memcached operation with only a single call. Before you
would have to ask for a specific connection and then do the operation.
This addition also includes exponential backoff code and handles
not my vbucket errors.

Change-Id: Ide2b91f1961470766853450b67485700c9e87fc9


COMMENTS

author: Kenny Maynard
date: 2012-05-02 03:58:31.720000000

Uploaded patch set 2.

-------------------------------------
author: Kenny Maynard
date: 2012-05-02 04:00:59.265000000

Patch Set 2: No score



-------------------------------------
author: Zackery Barr
date: 2012-05-02 04:52:05.915000000

Patch Set 2: (4 inline comments)

looks good. few minor comments.

Line:60, lib/memcached/helper/kvstore.py -> num_valid can be counter which is incremented every time  we do a set otherwise everytime we can len we are going to lock the kvstore 16 times

Line:463, lib/tasks/task.py -> not sure if its a good idea to let' this class acquire a partition lock. doesn't this mean that during verification we can't insert or update other rows in the same partition ?

Line:469, lib/tasks/task.py -> would this work with blobs ?

Line:819, lib/memcached/helper/data_helper.py -> let's not remove these two classes.  we can mark it as deprecated so we dont use this anymore for new test cases.  this is an attempt to just make the merge easier

-------------------------------------
author: Kenny Maynard
date: 2012-05-02 05:01:35.861000000

Patch Set 2: (4 inline comments)



Line:60, lib/memcached/helper/kvstore.py -> I will change this.

Line:463, lib/tasks/task.py -> Yes, but this is how the design works. We only lock portions of the hash table to gain concurrency. We also release the lock to let others in after each check.

Line:469, lib/tasks/task.py -> Blob support is planned for a future commit. Should be done tomorrow. This is only intended to support documents.

Line:819, lib/memcached/helper/data_helper.py -> This was only used in the Tommie's document generator stuff so I think it's okay to remove it since nothing else should rely on it.

-------------------------------------
