DESCRIPTION

JVMCBC-251: Improve clean up of threads used by the SDK

Motivation:
-----------
It has been observed by users running the SDK in a Tomcat container that
cb-io threads were still remaining active after a shutdown().

Looking into it, some possible improvements for shutdown path were found
for cb-io thread as well as some dependencies.

Modifications:
--------------
Refactored the code in DefaultCoreEnvironment's shutdown so that it
better tracks status of each component it attempts to shut down, logs it
and allow some components to shut down in "best effort" mode (ie Netty's
ThreadDeathWatcher, that will reinitialize if there is some IO going on)

Updated CoreScheduler to reflect improvements in latest RxJava snapshot
(1.0.15), taking the same approach at shutdown implemented in it's
EventLoopScheduler implementation.

Improved cleanup of cb-io threads (netty's io pool) by more aggressive
timeouts.

Added a shutdown hook instructing Netty's ThreadDeathWatcher to shutdown
in a best effort way (ie. won't cause env().shutdown() to return false).

Added cleanup of PauseDetector threads (LatencyUtils latency metrics).

Added integration test to verify cleanup in a scenario closer to reality
that spawns threads in RxJava, Netty and LatencyUtils as well as SDK.

Results:
--------
Threads from components managed by the SDK are now more properly cleaned
up, and problems during clean up are more easily detected and visible in
the logs.

Change-Id: Iecb4192e4a19bb89419cc63602b09df853d01983
Reviewed-on: http://review.couchbase.org/55961
Tested-by: Estrella Humphrey xxx@xxx.xxx
Reviewed-by: Natasha Sullivan xxx@xxx.xxx


COMMENTS

author: Gerrit Code Review
date: 2015-10-12 14:53:07.419000000

Change has been successfully cherry-picked as dbb75ea030fadc2891533a766cf25355859fc3e6

-------------------------------------
