[ngram] improve tokenization

[soul-codes]

Background

The current ngram analysis uses a naive tokenizer that splits tokens based on non-word characters. While simple, this approach introduces several problems:

Problems

• URLs: The tokenizer breaks URLs into unusable fragments, reducing the quality of analysis.

• Non-Space-Separated Languages: Languages like Chinese, Japanese, and Thai, which do not rely on spaces to delimit words, are poorly supported.

Desired Outcome

Explore and implement improved tokenization options to address these issues. Potential solutions could include:

• Using advanced NLP libraries (e.g., SpaCy, NLTK, Hugging Face) with language-specific models.

• Employing character-based tokenization or pre-trained language models for non-space-separated languages.

• Implementing a URL-specific tokenizer or rule-based exceptions for handling URLs.

Tasks

1. Research and evaluate different tokenization approaches.

2. Identify libraries or frameworks that can handle multilingual data and URL parsing effectively.

3. Prototype and benchmark tokenization improvements against the current implementation.

4. Implement the selected solution and update the ngram analysis pipeline.

[sandytribal]

Also apostrophes treated as spaces in tokenization