Steps to Reproduce

1. Install tinyld in a Vite project.
2. Run npm run build.

Expected Behavior

The build should complete successfully without errors.

Actual Behavior

The build fails with the above-mentioned error due to the unexpected character in tinyld.

Additional Context

The issue seems to be related to the handling of non-JavaScript files or unexpected characters in the tinyld module during the build process with Rollup.

Environment

• Vite version: 5.2.13
• Node version: v18.20.3
• tinyld version: v1.3.4

Vite config

`error during build:
node_modules/tinyld/dist/tinyld.normal.node.mjs (1:528704): Unexpected character '\u{200c}' (Note that you need plugins to import files that are not JavaScript)
file: /Users/zhovnir/Documents/GitHub/vite-diglot-ai/node_modules/tinyld/dist/tinyld.normal.node.mjs:1:528704

1: var N={TINYLD_CONFIG:"normal"};var y=/[,.。，、#%&/\ *¡!¿?[]！？;:…„“«»”"“_–—~]/gi,p=/[0-9]/g,S=/[\uFF10-\uFF19]/g,v=/\s\s /g,j=/’/gi,H=/[\u0300-\u036f]/g;function X(a){return typeof a=="string"||a instanceof String}function V(a){return a.toLowerCase().replace(j,"'").replace(y," ").replace(S,i=>String.fromCharCode(i.charCodeAt(0)-65248)).replace(p,"").replace(v," ").trim()}function U(a){return a.normalize("NFD").replace(H,"")}var P={light:{TRAINING_UNIQUE_GRAMS:... and lines more`

Thank you for your assistance!

Hello, yes I remember a similar issue like 2 years ago. At the time it was a special UTF-8 character not supported by esbuild minification. And after reporting it to esbuild, it was fixed under 48h

Unexpected character '\u{200c}

One of the temporary workaround could be to use vite.rollup.external settings, to exclude this library from processing.
I don't think we can help much here, it's totally outside of our scope.

It's one of the downside of a library like tinyld, the dictionary cover many languages and includes a lot of utf8 characters that are not so usual in codebase.