Thanks for suggesting. Could you please explain your desired outcome? "Customizing to fit different use cases" is too broad to be actionable. Also, customizing the search tokenizer for different languages can be implemented by using different separators for two different sites. In the discussion you linked, you mentioned:

For example on the mkdocs-material site if you search for test::search I'd expect it to return results about the two tokens (test and search) assuming the regex separator was applied to the query.

This was reported and fixed in #4884 and released in 9.0.7. The functionality for search transformation was removed because the current approach was not general enough and did not scale well. I'm happy to add a new way extension or transformation hook that allows to intercept query (and maybe results) and alter them before returning them, but we need to collect some use cases before tackling that.

Could you please explain your desired outcome?

@squidfunk sure: be able to modify the lunr.js pipeline to fit other use cases not covered by the default search pipeline - like this one: Two stage tokenization to add full strings to the index.

customizing the search tokenizer for different languages can be implemented by using different separators for two different sites

@squidfunk I don't think you can add tokens found that are separated by spaces and tokens found that follow some separator regex (two-stage tokenization) using the current separator approach, right?

I'm happy to add a new way extension or transformation hook that allows to intercept query (and maybe results)

@squidfunk I'm confused by this statement. This is the "old way" of doing it. The "new way" would be to take advantage of the features that lunr.js provides (pipelines)...right? Not sure of the effort involved to enable us to customize our own pipelines, but that is what this issue is asking for.

I'm confused by this statement. This is the "old way" of doing it. The "new way" would be to take advantage of the features that lunr.js provides (pipelines)...right? Not sure of the effort involved to enable us to customize our own pipelines, but that is what this issue is asking for.

I may have formulated a bit badly – yes, it's the old way, but I was talking about rethinking that process, as it only allowed for query transformation and nothing else. Transformation is now done as part of the worker, so maybe we could provide hooks to hook into different parts of the search index, possibly exposing one hook to alter Lunr.js before starting to index documents. This would effectively allow to implement own pipeline functions with which you should achieve what you we're aiming for when we talked about the two-stage tokenization approach.

To expand on that: we moved transformation into the worker, so the worker is completely self-contained, i.e., defines all behavior. This makes integration of third-party search solutions simpler, as the application itself will apply no processing to the query before sending it to the worker. Before, query transformation was done in the application, then sent to the worker.

All of this made the current approach unfeasible, since it involves defining a function in the global scope that is called by the application if defined. We need a new approach for search transformation / extension, but before I started working on that I wanted to verify that this is still something that is needed 😊 We'll add it back shortly. If you have other ideas that we should consider and requirements we need to fulfill, please share them here. So far we collected:

• Transform the query before searching (e.g. fooBar -> foobar foo bar)
• Register custom pipeline functions for expanding or filtering tokens before indexing
• Add a "Hello World" guide on how to write a custom pipeline function

Transform the query before searching (e.g. fooBar -> foobar foo bar)

@squidfunk so according to the lunr.js docs I think pipelines do this as well, no?

From the lunr.js docs:

lunr.Pipelines maintain an ordered list of functions to be applied to all tokens in documents entering the search index and queries being ran against the index.

So I think only point 2 and 3 would need to be implemented:

• Register custom pipeline functions for expanding or filtering tokens before indexing
• Add a "Hello World" guide on how to write a custom pipeline function

And point 3 you might just be able to link to the lunrjs docs like you do for the pymdownx stuff

I'm not sure if pipelines allow to change the entirety of the syntax, that is Lunr.js field references and operators for boosting, as well as inclusion and exclusion. I think pipelines will only allow to remove, replace, expand or add tokens. Thus, I believe that in the following query, only the terms in brackets are moved through the pipeline:

 title:[fooBar]* [fooBar]^2

This would not allow to split/replace meta characters or introduce additional prefix or suffix wildcards. However, more research is needed. If you wish to dig into this, it'll be awesome to get some intel. Otherwise, I'll do that later.

Please see the announcement in #6307.

I've reopened #6632 which specifically requests to make PascalCase searchable as PascalCase, pascalcase and case – a shortcoming of the current implementation that was reported several times. I'm confident that this will make it into the next iteration of search, as I was able to quickly throw together a prototype. I'm leaving this issue open, since we also want to allow users to easily change the behavior of search with custom hooks.