This pull request introduces enhancements to the DataCollatorForLanguageModeling class, providing greater flexibility for token replacement during masked language modeling (MLM). The key changes include:

1. Configurable Replacement Probabilities:

   • mask_replace_prob: Specifies the probability of replacing masked tokens with the [MASK] token (default: 80%).
   • random_replace_prob: Specifies the probability of replacing masked tokens with random tokens from the vocabulary (default: 10%).
   • The remaining masked tokens are left unchanged (default: 10%).

2. Edge Case Handling:

   • Properly scales random_replace_prob to the remaining probability after applying mask_replace_prob.
   • Includes validation to ensure the sum of mask_replace_prob and random_replace_prob does not exceed 1.

3. Backward Compatibility:

   • Default behavior mimics the traditional 80-10-10 rule for MLM token replacement.

Examples of New Functionality

• Default Behavior:
  Replace 80% of masked tokens with [MASK], 10% with random tokens, and leave 10% unchanged.
• Custom Configurations:
  • Replace all masked tokens with [MASK]:

    mask_replace_prob=1.0, random_replace_prob=0.0

  • Replace all masked tokens with random tokens:

    mask_replace_prob=0.0, random_replace_prob=1.0

  • Balanced replacement:

    mask_replace_prob=0.5, random_replace_prob=0.4

Additional Notes

• Updated docstrings to reflect the new configuration options.
• Added validations for probability values and enhanced edge case handling for robust training workflows.

This enhancement gives users greater control over MLM training configurations, catering to various pretraining and fine-tuning use cases.