DESCRIPTION

SPY-180: Retry auth ops on error and timeouts.

Motivation
----------
If a very short operation timeout is used, or any of the auth operations
fail during their execution, the whole auth process can be "messed up" and
nodes will take longer to authenticate to the proper node, or fail to do so
completely.

Modifications
-------------
This changeset consists of two modifications:

  - The AuthThread has been made resilient for all three types of operations
    (sasl list, sasl auth and sasl step) with a configurable timeout. If this
    is exceeded, or the operation fails for any other reason, it is retried.

  - The TCPMemcachedNodeImpl has been modified to not cancel an auth operation
    if the default op timeout exceeds before writing it onto the network. The
    actual sasl operations should only be subject to their own timeouts and
    retry if needed. This avoids failing auth if a very short timeout is used
    for regular operations.

Result
------
Authentication, both with regular and short op timeouts, are much more reliable
and failing op steps are now retried transparently (although logged for further
debugging purposes). The default step timeout is set to 2500ms to be aligned with
the default op timeout, but can be changed through the "net.spy.authOpTimeout"
system property.

Change-Id: I2721a3053dc0f091783183048b8ab4872d0ae823


COMMENTS

author: Natasha Sullivan
date: 2014-12-10 12:08:29.193000000

Uploaded patch set 2.

-------------------------------------
author: Natasha Sullivan
date: 2014-12-10 12:10:35.657000000

Patch Set 2:

I wonder if I should add unit tests here, but it will be a pain to do because of all the injected dependencies.

-------------------------------------
author: Estrella Humphrey
date: 2014-12-10 14:02:35.494000000

Patch Set 2:

(1 comment)

Line:118, src/main/java/net/spy/memcached/auth/AuthThread.java -> shouldn't the loop exit in case of done set to true?

-------------------------------------
