Full text search (in Japanese) #248
Description
Not necessarily part of this project, but I was thinking, to make Cloudflare D1 viable for CJK projects, I would want to have some way to search for CJK text too. As SQLite official FTS does not support tokenizing Japanese well, it may be interesting to implement my own tokenizer in JavaScript to support this. (Can Intl.Segmenter be used in CF Workers?) or alternatively, perhaps we would just need to do a prepass on the Japanese text before passing to SQLite FTS.