DESCRIPTION

CBQE-3534 - Add some more tests for custom analyzer functionality

Change-Id: I1a85bb5af68ec466479d21b1b4c8cb4c16dc3177


COMMENTS

author: Alissa Huang
date: 2016-08-29 18:18:00.173000000

Uploaded patch set 1.

-------------------------------------
author: Angelina Mayer
date: 2016-08-30 23:32:40.650000000

Patch Set 1: Code-Review+1

(6 comments)

mostly ok, just some minor qns.

Line:124, pytests/fts/custom_map_generator/map_generator.py -> is it the same list of stopwords supported by Bleve? I have the same copied to es_base.py to create a custom analyzer on ES with the bleve stop words so we can compare results for standard analyzer.

Line:55, pytests/fts/es_base.py -> what does this mean?

Line:713, pytests/fts/fts_base.py -> maybe leave a comment that you are doing a deepcopy of child_map only because child map and field map have same fields?

Line:503, pytests/fts/stable_topology_fts.py -> fr wiki?

Line:512, pytests/fts/stable_topology_fts.py -> i think* I saw these in last review of another test as well. If I'm not wrong, maybe these can be moved to fts_base.py?

Line:514, pytests/fts/stable_topology_fts.py -> I don't understand this part. I don't see lang=FR being passed(you are loading EN wiki) so why is it that "Republic" does not show up with en analyzer but FR?

-------------------------------------
author: Alissa Huang
date: 2016-08-31 05:24:41.287000000

Patch Set 1:

(6 comments)

Line:124, pytests/fts/custom_map_generator/map_generator.py -> Yes, I have created custom analyzers on Bleve and ES with the same map for stopwords so that we get identical results. No need of putting it in a file on ES, we can just pass the list in the index definitions.

Line:55, pytests/fts/es_base.py -> This char filter will replace all occurences of characters 'f' with 'ph'. Eg. 'fone' will become 'phone'.

Line:713, pytests/fts/fts_base.py -> Done

Line:503, pytests/fts/stable_topology_fts.py -> No this is the normal wiki dataset, I havent specified the language. The revision.text.#text field has multi-lingual data, so I thought it would be best to use that field for this test.

Line:512, pytests/fts/stable_topology_fts.py -> Done

Line:514, pytests/fts/stable_topology_fts.py -> Yes, this is the default wiki dataset (I believe its EN). Somehow with the EN analyzer, Republic doesn't show up, but it does with FR analyzer. That is not correct, maybe I can change the query or the analyzer for this test. But atleast here, we are able to test co-existence of two analyzers for the same field. I will investigate more on the best data for this test and update it in the next commit.

-------------------------------------
author: Tate Garrett
date: 2016-08-31 05:26:07.306000000

Change has been successfully cherry-picked as 55b70c9d9efaa55842b77966a75f9561006e68ab by Alissa Huang

-------------------------------------
