DESCRIPTION

Fix concurrent access to operations objects, especially near timeouts.

We discovered that having one thread call get() on futures with a
timeout much shorter than the timeout built into the client could
(after a while) reliably kill off the IO thread with a
NullPointerException. After restarting with assertions enabled, we
found many other ways to make the IO thread die relating mostly to
operations that were canceled or timed out in one thread while they
were being used in another thread.

The solution to this is to make "timing out" an operation something
similar to "cancel" - that is, not a state of the operation's state
machine, but just a flag. Then, since this flag (as with the cancelled
flag) can be read and set from multiple threads, synchronize the
methods that do that.

Ideally, transitionState and getState would never be called outside
the IO thread; however, since at least getState is currently, both
getState and transitionState should then also be synchronized.

With these changes, we don't end up killing off the IO thread anymore.

Change-Id: I280f31ed47922e27da3d5dc82f640c114d6014d7


COMMENTS

author: Jordyn Hooper
date: 2011-09-06 14:13:10.299000000

Uploaded patch set 2.

-------------------------------------
author: Jordyn Hooper
date: 2011-09-06 15:11:22.741000000

Patch Set 2: (3 inline comments)

And in doing a bit of self-review, I see I made an error and will need to upload patchset 3...

Line:191, src/main/java/net/spy/memcached/protocol/TCPMemcachedNodeImpl.java -> Note that it is possible - though rare - at this point for the operation to be timed out or cancelled, since there's no way to acquire a synchronization lock in getNextWritableOp and still be holding the same lock here. Therefore, the operations in this loop body need to not fail when dealing with timed out/canceled ops. (among other things, this means timing out an operation can't null out the write buffer)

Line:186, src/main/java/net/spy/memcached/protocol/BaseOperationImpl.java -> I'd like to either fix this assertion to not a tautology or remove it. What should it be? What's the intent?

Line:203, src/main/java/net/spy/memcached/protocol/BaseOperationImpl.java -> ditto

-------------------------------------
