-
Notifications
You must be signed in to change notification settings - Fork 1
dowobeha/europarl_tools
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Europarl v6 Preprocessing Tools =============================== written by Philipp Koehn and Josh Schroeder Sentence Splitter ================= Usage ./split-sentences.perl -l [en|de|...] < textfile > splitfile Uses punctuation and Capitalization clues to split paragraphs of sentences into files with one sentence per line. For example: This is a paragraph. It contains several sentences. "But why," you ask? goes to: This is a paragraph. It contains several sentences. "But why," you ask? See more information in the Nonbreaking Prefixes section. Nonbreaking Prefixes Directory ============================== Nonbreaking prefixes are loosely defined as any word ending in a period that does NOT indicate an end of sentence marker. A basic example is Mr. and Ms. in English. The sentence splitter and tokenizer included with this release both use the nonbreaking prefix files included in this directory. To add a file for other languages, follow the naming convention nonbreaking_prefix.?? and use the two-letter language code you intend to use when calling split-sentences.perl and tokenizer.perl. Both split-sentences and tokenizer will first look for a file for the language they are processing, and fall back to English if a file for that language is not found. If the nonbreaking_prefixes directory does not exist at the same location as the split-sentences.perl and tokenizer.perl files, they will not run. For the splitter, normally a period followed by an uppercase word results in a sentence split. If the word preceeding the period is a nonbreaking prefix, this line break is not inserted. For the tokenizer, a nonbreaking prefix is not separated from its period with a space. A special case of prefixes, NUMERIC_ONLY, is included for special cases where the prefix should be handled ONLY when before numbers. For example, "Article No. 24 states this." the No. is a nonbreaking prefix. However, in "No. It is not true." No functions as a word. See the example prefix files included here for more examples.
About
No description, website, or topics provided.
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published