DESCRIPTION

JVMCBC-436: Allow keepalives to recycle dead connections.

Motivation
----------
We've been sending the app-level keepalives to each service
to make sure that inactive sockets are not cut off by firewalls
or other networking equipment.

To make the system more reliable we can also take the concept
of the keepalives and enhance it a bit more to act more like
"heartbeats".

Modifications
-------------
The keepalive code, alongside with its configurables on the
environment, has been extended to support the following
functionality:

  1) send the keepalives continuously, not just when the
     socket is idle.
  2) give the keepalives a timeout so that we can measure
     if they take too long to return.
  3) set a threshold that if N of these keepalives return
     with an error (cancelled, timed out) the socket will
     be recycled as a proactive measure.

All these settings are configurable and one "good"
keepalive sets the threshold counter back to 0, all these
counters are on a per socket basis.

Result
------
This changeset makes potentially dead/half-dead tcp connections
surface more quickly, recycle them and as a result making the
whole syste more robust to external misbehaving network
equipment.

Change-Id: Ica5128f0d07a058ac5c7dcf6d77f60490bbb4bea


COMMENTS

author: Marques Ellis
date: 2017-07-10 21:52:02.931000000

Patch Set 6: Patch Set 5 was rebased

-------------------------------------
author: Marques Ellis
date: 2017-07-10 21:52:06.232000000

Patch Set 6: Verified+1

-------------------------------------
