DESCRIPTION

CBQE-3067 Add/organize fts tests, add in fts random query generator for testing queries

Change-Id: If044f4a46e1eef5a8b1a9d96b8a0faae0f219390


COMMENTS

author: Hugo Blankenship
date: 2015-11-07 02:56:12.885000000

Patch Set 4:

Build Started http://factory.couchbase.com/job/testrunner-Jasmin Rangel-sherlock/6106/

-------------------------------------
author: Hugo Blankenship
date: 2015-11-07 03:25:09.120000000

Patch Set 4: Verified+1

Build Successful 

http://factory.couchbase.com/job/testrunner-Jasmin Rangel-sherlock/6106/ : SUCCESS

-------------------------------------
author: Angelina Mayer
date: 2015-11-09 19:21:32.615000000

Patch Set 5: Patch Set 4 was rebased

-------------------------------------
author: Nico George
date: 2015-11-09 19:47:04.333000000

Patch Set 4:

(17 comments)

Line:28, pytests/fts/emp_rand_query_gen.py -> can we have another class with fields and query_types list . Then I can inherit those class if i miss anything

Line:36, pytests/fts/emp_rand_query_gen.py -> need a check if query_types is None

Line:52, pytests/fts/emp_rand_query_gen.py -> only 2 or 3 ? any reason ?

Line:69, pytests/fts/emp_rand_query_gen.py -> -ve case what if both time it returns 0L or 1L . So its almost predictable , we can have two sets of queries .  

1 set with must , other with not must

Line:119, pytests/fts/emp_rand_query_gen.py -> for numeric query add two extreme cases , how it will behave for MAXINT and MININT , is search happening .  so for any value we can have three queries . lets value be x , range [MININT x] , [ x x] , [ x MAXINT]

Line:145, pytests/fts/emp_rand_query_gen.py -> instead of bool(random.getrandbits(1))  , why don't we take it from the list of different queries , or we can maintain a cache for queries .  for complex queries , check the cache , which queries are created and randomly pick 3-4 and create new query

Line:157, pytests/fts/emp_rand_query_gen.py -> don't use list , use SET , what if any duplicate ?

Line:240, pytests/fts/emp_rand_query_gen.py -> we need main ?

Line:31, pytests/fts/stable_topology_fts.py -> where is query_types?

Line:2067, pytests/fts/fts_base.py -> we can have another config file where different dataset is mentioned , what if in future we have verizon dataset

Line:2240, pytests/fts/fts_base.py -> will it be enhanced or is it like this ?

Line:2254, pytests/fts/fts_base.py -> can we avoid json_dump here , we are doing it few times in query generator. return the json format

Line:2255, pytests/fts/fts_base.py -> we can store it in class , which can be reused

Line:2258, pytests/fts/fts_base.py -> whats difference with this procedure with the above ? i mean when we have random_query , why we need construct query ?

Line:2268, pytests/fts/fts_base.py -> are you sure that hits will be 0 for unsiccessful query ?  what if it returns NULL . code will break

Line:2295, pytests/fts/fts_base.py -> don't use XRANGE , its discarded in python 3

Line:28, lib/couchbase_helper/data.py -> for LAST_NAMES can you include like  Sr. , senior . If the name is george bush senior or george bush sr.  / besides some historical names like richard VIII , or richard IX . and and names with utf8 character ? like spanish or german names

-------------------------------------
author: Angelina Mayer
date: 2015-11-10 23:37:29.567000000

Patch Set 4:

(16 comments)

Line:28, pytests/fts/emp_rand_query_gen.py -> Yes, we can, I will add.

Line:36, pytests/fts/emp_rand_query_gen.py -> It's usually never none, by default it's ["match"].

Line:52, pytests/fts/emp_rand_query_gen.py -> for now, just testing a complex query upto 3 match queries. When we are ready for more complex queries, we can change later.

Line:69, pytests/fts/emp_rand_query_gen.py -> discussed.

Line:145, pytests/fts/emp_rand_query_gen.py -> there is no concept of cache in testrunner with every test being a separate invocation of testrunner. Generation involves zero overhead on the other hand. Also with the concept of seeding, every test will have its own set of repeatable queries.

Line:157, pytests/fts/emp_rand_query_gen.py -> this returns only one random value from a list. We are not populating list, only retrieving values.

Line:240, pytests/fts/emp_rand_query_gen.py -> not needed but good to have, to run as a standalone script say to generate valid bleve queries. Easier to test and enhance.

Line:31, pytests/fts/stable_topology_fts.py -> defined in FTSBaseTest class.

Line:2067, pytests/fts/fts_base.py -> discussed

Line:2240, pytests/fts/fts_base.py -> We will most probably only change the above 2 values as plan params. No more enhancement planned for this method.
self.index_replicas	and self.partitions_per_pindex are passed in the test as cmd line params.

Line:2254, pytests/fts/fts_base.py -> I see json.dumps used only in 2 places in query generator for the same reason.

json.dumps is used to replace some patterns in the dict with another. You are right that i need to return json, perhaps i can use json.loads()

Line:2255, pytests/fts/fts_base.py -> good idea, done

Line:2258, pytests/fts/fts_base.py -> Random query generator generates a valid Bleve query. construct_query_json encloses the bleve query in a json format fts understands. The format is different - see QUERY.JSON class.

Line:2268, pytests/fts/fts_base.py -> this is the same value as 'x' in "x matches found" for any query. If null is returned for any case, it's a bug and code should break to bring it to our attention.

Line:2295, pytests/fts/fts_base.py -> Good catch. All our slaves are on python2.7. range() is not memory efficient in <  python 3. However this is a small list so I will use range()

Line:28, lib/couchbase_helper/data.py -> Spanish, german names - good idea, adding.

-------------------------------------
author: Tate Garrett
date: 2015-11-11 23:45:14.909000000

Change has been successfully cherry-picked as 75ff382876ff5c453d7550a3ef245cd94c86999a

-------------------------------------
