TextCritical.net

Could also add a button to convert the beta-code to Unicode

I think I may be able to make a stemming function that can help perform searches with beta-code.

• http://pythonhosted.org/Whoosh/stemming.html?highlight=stemming
• http://stackoverflow.com/questions/3688432/djangohaystackwhoosh-how-to-deal-with-language-inflection
• http://pythonhosted.org/Whoosh/api/analysis.html?highlight=stemming#whoosh.analysis.StemmingAnalyzer

Actually, I may want to use variations: http://pythonhosted.org/Whoosh/api/query.html#whoosh.query.Variations

I see information about how the Tokenizers split up the content for searching but I cannot find docs about how the searches are split up.

I think the SpaceSeparatedTokenizer is the correct one to use.

This person seems to be running into the same problem: http://stackoverflow.com/questions/14296792/haystack-whoosh-search-for-email-address-with-at-symbol/14482122#14482122

Asked question on StackOverflow: http://stackoverflow.com/questions/14932013/include-slashes-and-parentheses-in-tokens

It looks like the query parser uses the analyzer associated with the schema to parse the search. See QueryParser.__init__:

:param schema: a :class:`whoosh.fields.Schema` object to use when
parsing. The appropriate fields in the schema will be used to
tokenize terms/phrases before they are turned into query objects.
You can specify None for the schema to create a parser that does
not analyze the text of the query, usually for testing purposes.

Changing the regular expression that SimpleAnalyzer uses to "[\w/*()=\+|&']+(\.?\w+)*" does get the unit tests to pass. However, the searches still fail. I'm guessing that the analyzer gets persisted in the index file and thus the indexes have to be re-created.

Re-indexing the database didn't seem to help. Doing a search for "pa/sxa" returns no results despite the fact that this is converted to "πάσχα". Oddly enough, searching for "πάσχα" does return results. These are actually different:

CF 80 E1 BD B1 CF 83 CF 87 CE B1
CF 80 CE AC    CF 83 CF 87 CE B1

Somehow, the unicode for the accent is getting saved differently.

Normalizing the content before saving it in the index worked. The only problem is that highlighting doesn't happen.

The analyzer does seem to be pickled. I'll need to recreate the indexes from scratch.

I rebuilt the indexes. The search works provided I normalize the unicode on the search term. The problem is that the terms are not being highlighted, either with a search for πάσχα or pa/sxa.

Hmm, not sure what is going on here. Highlighting is inconsistent. ἑορτὴ does get highlighted.

Somehow the content is getting stored in a different format in the indexes. I normalized the content before I provide it to the highlights function and this fixed the problem.

Nice, it looks like the beta-code causes the text to get highlighted correctly now. Yay