DESCRIPTION

add hot shift

Change-Id: Ie137d6cb466a3fa286530eb41f1158b2313f8169


COMMENTS

author: Alex Mcmahon
date: 2012-05-08 22:42:32.012000000

Patch Set 1: Looks good to me, but someone else must approve



-------------------------------------
author: Hugo Blankenship
date: 2012-05-08 22:49:16.848000000

Patch Set 1:

Build Started http://qa.hq.northscale.net/job/testrunner-Jasmin Rangel-master/19/ 

-------------------------------------
author: Hugo Blankenship
date: 2012-05-08 22:54:40.104000000

Patch Set 1: Verified

Build Successful 

http://qa.hq.northscale.net/job/testrunner-Jasmin Rangel-master/19/ : SUCCESS

-------------------------------------
author: Noah Dean
date: 2012-05-08 22:57:33.397000000

Patch Set 1: (2 inline comments)

Seems like writing the config in python would save you guys *tons* of work and prevent copy and paste stuff.  And you'd have an obvious way to include config snippets, build generators, etc...

Line:490, pytests/performance/perf.py -> This line looks kind of bizarre.  Why is the boolean being converted to an int?

Line:497, pytests/performance/perf.py -> What is hot_shift?  Might be good to describe this a bit better.

-------------------------------------
author: Nora Conway
date: 2012-05-08 23:25:17.675000000

Patch Set 1: Looks good to me, approved

(2 inline comments)

On using python to hold config for all these performance tests, we're actually going the other way.  That is, we'd tried using python as our declarative config DSL (seemed like a good idea), but nowadays we've been trying to use less python and more config files, for a few reasons...

- explicitness of the actual run parameters
- the python had become a morass of copy & paste code
- separate config files let us reach closer to a policy of immutability of test run input parameters, so that we can always reproduce a run just knowing the config file name, product version, cluster name, and testrunner version.

On the config files, they're in testrunner/conf/**/*.conf

Line:490, pytests/performance/perf.py -> Because mcsoda takes an int here -- I wasn't pythonistic enough.

Line:497, pytests/performance/perf.py -> This was one of those legit bugs where the function in the middle wasn't passing along a parameter.

On hot-shift, please see...

https://github.com/couchbase/testrunner/blob/master/pytests/performance/mcsoda.py#L1142

-------------------------------------
author: Nora Conway
date: 2012-05-08 23:25:18.095000000

Change has been successfully cherry-picked as e4eb9232416844eb050256420c9c355a0211a872.

-------------------------------------
author: Noah Dean
date: 2012-05-08 23:43:20.742000000

Patch Set 1:

Your second and third goals are mostly in conflict.  I agree with avoiding copy and paste, but you could at least avoid it selectively by reading the config with "execfile".  You're already doing that, but you're having to line config file changes up with code like this:  https://github.com/couchbase/testrunner/blob/master/pytests/upgradetests.py#L406 -- surely you don't think that's easier to maintain.

I don't think you're going to get repeatable tests on changing code just because the configuration isn't changing.  The code is changing around it, so the easiest and most reliable way to repeat a test is to check out the code from when it ran and reuse the same config anyway.

-------------------------------------
