DESCRIPTION

Fix concurrent access to operations objects, especially near timeouts

We discovered that having one thread call get() on futures with a
timeout much shorter than the timeout built into the client could
(after a while) reliably kill off the IO thread with a
NullPointerException. After restarting with assertions enabled, we
found many other ways to make the IO thread die relating mostly to
operations that were canceled or timed out in one thread while they
were being used in another thread.

The solution to this is to make "timing out" an operation something
similar to "cancel" - that is, not a state of the operation's state
machine, but just a flag. Then, since this flag (as with the cancelled
flag) can be read and set from multiple threads, synchronize the
methods that do that.

Ideally, transitionState and getState would never be called outside
the IO thread; however, since at least getState is currently, both
getState and transitionState should then also be synchronized.

With these changes, we don't end up killing off the IO thread anymore.

Change-Id: Ice7bac2ba1e2dbcd2004f5d7920944eef6bbbcb8


COMMENTS

author: Kenny Maynard
date: 2011-09-06 23:47:00.783000000

Patch Set 1: Verified; Looks good to me, approved



-------------------------------------
author: Kenny Maynard
date: 2011-09-06 23:47:02.862000000

Change has been successfully cherry-picked as a9035c9e8243aec93d9516c88f33f6e62d4020c7.

-------------------------------------
