67 Pull requests merged by 24 people

• Bump Dockerfile.ci (2024-10-05)

  #10776 merged Oct 5, 2024

• Fix load nemo issue in Multimodal TRTLLM

  #10744 merged Oct 5, 2024

• ci: Fix broken notifications

  #10774 merged Oct 5, 2024

• ci: Re-instantiate optional tests

  #10739 merged Oct 5, 2024

• Bump Dockerfile.ci (2024-10-05)

  #10768 merged Oct 5, 2024

• Checkpoint connector bugfixes

  #10647 merged Oct 5, 2024

• SP=off

  #10753 merged Oct 4, 2024

• Add NEST SSL to main

  #10319 merged Oct 4, 2024

• Akoumparouli/nemo ux test fixes (#10641)

  #10761 merged Oct 4, 2024

• Packed Sequence [NeMo 2]

  #10445 merged Oct 4, 2024

• Cherry pick Vllm 0.6.0 integration test (10697) into r2.0.0

  #10760 merged Oct 4, 2024

• add diffusion energon dataloader

  #10700 merged Oct 4, 2024

• Revert always_save_context to False

  #10667 merged Oct 4, 2024

• ci: Fix Nemo_CICD_Test step

  #10756 merged Oct 4, 2024

• Vllm 0.6.0 integration test

  #10697 merged Oct 4, 2024

• Cherry pick Remove finetuning recipes for Long Context (10703) into r2.0.0

  #10746 merged Oct 4, 2024

• Akoumparouli/nemo ux test fixes

  #10641 merged Oct 4, 2024

• Bump Dockerfile.ci (2024-10-04)

  #10754 merged Oct 4, 2024

• Cherry pick Fix typo in ASR RNNT BPE model (10742) into r2.0.0

  #10743 merged Oct 4, 2024

• Fix semi_sorted paper link

  #9597 merged Oct 4, 2024

• Cherry pick Update NeVA Mixtral Tutorial (10669) into r2.0.0

  #10705 merged Oct 4, 2024

• ci: Improve caching and image retention

  #10735 merged Oct 4, 2024

• Remove long context recipe finetuning tests

  #10755 merged Oct 4, 2024

• ci: Apply hotfix to Nemo_CICD_Test

  #10757 merged Oct 4, 2024

• Cherry pick In-framework inference fixes (10698) into r2.0.0

  #10731 merged Oct 4, 2024

• Add Llama 3.1 Pruning and Distillation Tutorial

  #10720 merged Oct 4, 2024

• TestEncDecMultiTaskModel for canary parallel

  #10740 merged Oct 4, 2024

• Update to PT24.07 in Dockerfile.ci

  #10358 merged Oct 4, 2024

• remove rm -rf /home/TestData and use /tmp instead (#10729)

  #10751 merged Oct 3, 2024

• Cherry pick Akoumparouli/fix get tokenizer list (10596) into r2.0.0

  #10684 merged Oct 3, 2024

• ci(slack): Fix job pagination

  #10737 merged Oct 3, 2024

• Cherrypick 10457

  #10726 merged Oct 3, 2024

• Cherrypick 10470

  #10724 merged Oct 3, 2024

• Cherrypick #10498

  #10723 merged Oct 3, 2024

• Akoumparouli/nemo ux cherrypick 10361 & 10441 & 10423

  #10526 merged Oct 3, 2024

• Akoumparouli/nemo ux cherrypick 10363

  #10587 merged Oct 3, 2024

• Remove finetuning recipes for Long Context

  #10703 merged Oct 3, 2024

• remove rm -rf /home/TestData and use /tmp instead

  #10729 merged Oct 3, 2024

• Fix typo in ASR RNNT BPE model

  #10742 merged Oct 3, 2024

• Bump Dockerfile.ci (2024-10-03)

  #10727 merged Oct 3, 2024

• ci: Fix secret

  #10736 merged Oct 3, 2024

• ci: Fix issue with feedback

  #10734 merged Oct 3, 2024

• ci: Add workflow for scheduled VM reboot

  #10695 merged Oct 3, 2024

• In-framework inference fixes

  #10698 merged Oct 3, 2024

• Cherrypick 10300

  #10722 merged Oct 2, 2024

• NeMo 2.0 mixtral ci test

  #10655 merged Oct 2, 2024

• feat: Migrate GPTSession refit path in Nemo export to ModelRunner for Aligner

  #10654 merged Oct 2, 2024

• [feat] Update get_model_parallel_src_rank to support tp-pp-dp ordering

  #10652 merged Oct 2, 2024

• [fix] Ensures disabling exp_manager with exp_manager=null does not error

  #10651 merged Oct 2, 2024

• ci: Restore docker cache

  #10708 merged Oct 2, 2024

• [McoreDistOptim] fix the naming to match apex.dist

  #10707 merged Oct 2, 2024

• ci: Disable feedback on forks

  #10709 merged Oct 2, 2024

• Add NeMo 2.0 section to the readme

  #10646 merged Oct 2, 2024

• Update NeVA Mixtral Tutorial

  #10669 merged Oct 1, 2024

• a few fixes for the new prompt template based dataloader and lora distributed fused adam

  #10701 merged Oct 1, 2024

• Multimodal conversation format dataloading

  #10683 merged Oct 1, 2024

• ci: Stability to CI/CD

  #10694 merged Oct 1, 2024

• Revert "Cherry pick Updating modelopt spec for Mixtral (10660) into r2.0.0"

  #10687 merged Oct 1, 2024

• Require setuptools>=70 and update deprecated api (#10659)

  #10686 merged Sep 30, 2024

• DB tutorial ckpt path update

  #10662 merged Sep 30, 2024

• Cherrypick #10466 #10611 #10632 without the ci test

  #10638 merged Sep 30, 2024

• [NeMo-UX] Support save_last="link"

  #10548 merged Sep 30, 2024

• Akoumparouli/fix get tokenizer list

  #10596 merged Sep 30, 2024

• Require setuptools>=70 and update deprecated api

  #10659 merged Sep 30, 2024

• Cherry pick Updating modelopt spec for Mixtral (10660) into r2.0.0

  #10664 merged Sep 30, 2024

• Cherry pick Fix asr warnings (10469) into r2.0.0

  #10636 merged Sep 30, 2024

• Cherry pick Fix Clip initializing issue in r2.0.0 (10585) into r2.0.0

  #10633 merged Sep 30, 2024

42 Pull requests opened by 28 people

• Fix typo in rnnt_bpe_models.py: tokenier -> tokenizer

  #10676 opened Sep 29, 2024

• Cherry pick `DB tutorial ckpt path update (10662)` into `r2.0.0`

  #10685 opened Sep 30, 2024

• Context Parallel SFT Support for dataset in THD format

  #10688 opened Sep 30, 2024

• DAPT with NeMo FW

  #10689 opened Sep 30, 2024

• Add assertion for always save nemo add model parallel size

  #10690 opened Sep 30, 2024

• mixtral bitexact ci test

  #10692 opened Oct 1, 2024

• MCore Partial DistOpt Feature

  #10693 opened Oct 1, 2024

• Cherry pick 10559 and 10555

  #10696 opened Oct 1, 2024

• Allow logging memory profile on interval

  #10699 opened Oct 1, 2024

• Mixtral set seq_length=4k

  #10704 opened Oct 1, 2024

• Support `tie_word_embeddings=True` in `convert_mistral_7b_nemo_to_hf.py`

  #10710 opened Oct 2, 2024

• Add a build option to load_context

  #10713 opened Oct 2, 2024

• Upt nemo2 ckpt inn TRT-LLM export

  #10714 opened Oct 2, 2024

• Adding NeMo 2.0 T5 finetuning (on Squad dataset)

  #10716 opened Oct 2, 2024

• DRAFT: Release/nim 24.08 no deploy trt llm only

  #10718 opened Oct 2, 2024

• use model device for embedding extraction

  #10719 opened Oct 2, 2024

• Fix gradient clipping mistral

  #10725 opened Oct 2, 2024

• Fix error raising logic in model import

  #10728 opened Oct 3, 2024

• Track io for hf AutoTokenizer

  #10730 opened Oct 3, 2024

• Moving steps to MegatronParallel to improve UX for Fabric

  #10732 opened Oct 3, 2024

• Adding init_model_parallel to FabricMegatronStrategy

  #10733 opened Oct 3, 2024

• allow cudnn fa

  #10741 opened Oct 3, 2024

• perf recipes

  #10747 opened Oct 3, 2024

• [MCoreDistOptim] Add assertions for McoreDistOptim and fix fp8 arg specs

  #10748 opened Oct 3, 2024

• Yash/dev llava next

  #10749 opened Oct 3, 2024

• [Draft] Fix optimizer state_dict compatibility between Apex's fused_adam and distributed_fused_adam

  #10750 opened Oct 3, 2024

• [Draft] Add flux inference pipeline

  #10752 opened Oct 4, 2024

• ci: Improve VM maintenance

  #10758 opened Oct 4, 2024

• multiturn training support for SALM

  #10759 opened Oct 4, 2024

• gpt20b, gpt175b ub configs

  #10762 opened Oct 4, 2024

• [DRAFT] Add LLama32 Vision Model Support in Nemo 2.0

  #10763 opened Oct 4, 2024

• remove 8x3b recipes

  #10764 opened Oct 4, 2024

• Save yaml config for model in nemo.lightning.io

  #10765 opened Oct 4, 2024

• disable dynamo for DDP test

  #10766 opened Oct 4, 2024

• Pagaray/reduce storage usage in favor of local storage

  #10767 opened Oct 4, 2024

• [Draft] Fix checkpoint loading when lm_head is on separate pipeline stage

  #10769 opened Oct 5, 2024

• Cherry pick `Checkpoint connector bugfixes (10647)` into `r2.0.0`

  #10770 opened Oct 5, 2024

• Dpykhtar/llama3 8b 24.09 train

  #10771 opened Oct 5, 2024

• ci: Cleanup required tests

  #10773 opened Oct 5, 2024

• ci: Add cherry-pick label to cherry-picks

  #10775 opened Oct 5, 2024

• replace `SIGKILL` with `SIGTERM`

  #10777 opened Oct 5, 2024

• Bump `Dockerfile.ci` (2024-10-06)

  #10778 opened Oct 6, 2024

11 Issues closed by 3 people

• NeMo/tutorials/speaker_tasks/ASR_with_SpeakerDiarization needs confidence estimation

  #10283 closed Oct 6, 2024

• slurm Multi-machine and multi-GPU training

  #10229 closed Oct 4, 2024

• Converting Script for Mamba2 Hybrid to HF/Pytorch

  #10268 closed Oct 4, 2024

• fix exp_manager.py to work on Windows

  #10275 closed Oct 4, 2024

• be prepared when a user selects a file that isn't strictly mono and other file extensions as well

  #10276 closed Oct 4, 2024

• fix where the transcription is saved please

  #10277 closed Oct 4, 2024

• MCore slower than NeMo native implementation

  #9524 closed Sep 30, 2024

• Question: Difference of paths2audio_files param and path in manifest file for speaker diarization with ClusteringDiarizer

  #9789 closed Sep 30, 2024

• AttributeError: 'MegatronGPTModel' object has no attribute 'decoder'

  #10034 closed Sep 30, 2024

• NeMo 2.0 llm sharded_state_dict error

  #10675 closed Sep 30, 2024

• Punctuation and Capitalization Model not working

  #10615 closed Sep 29, 2024

8 Issues opened by 7 people

• `IPython` should be included in the requirements

  #10772 opened Oct 5, 2024

• Loading 70B model from .nemo checkpoint takes very long time

  #10745 opened Oct 3, 2024

• How to implement weight decay towards the pre-trained model?

  #10738 opened Oct 3, 2024

• Global shape mismatch for loaded ((1024, 768)) and expected ((512, 768)) tensor for key model.embedding.position_embeddings.weight

  #10715 opened Oct 2, 2024

• Using MSDD model with a different speaker embedding model

  #10681 opened Sep 30, 2024

• Unable to decode using canary 1b model

  #10680 opened Sep 30, 2024

• When converting a checkpoint from Hugging Face, the checkpoint format conversion keeps getting CUDA out of memory

  #10679 opened Sep 30, 2024

• Punctuation and Capitalization Model: how to add custom Punctuation marks to prepare data script?

  #10677 opened Sep 29, 2024

59 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Low-frame rate Speech Codec Implementation

  #10298 commented on Oct 2, 2024 • 9 new comments

• Random read for tarr files in lhotse dataloaders

  #10536 commented on Oct 3, 2024 • 6 new comments

• Flashlight and Pyctcdecode decoders

  #8428 commented on Oct 2, 2024 • 5 new comments

• Use torch sdpa implementation in ASR mha

  #9590 commented on Oct 5, 2024 • 4 new comments

• Disable checkpoint conversion inside AutoResume

  #10645 commented on Oct 5, 2024 • 3 new comments

• PTQ example for NeMo 2.0

  #10642 commented on Oct 4, 2024 • 3 new comments

• [NeMo-UX] Add llm.generate to nemo.collections.llm

  #10471 commented on Oct 2, 2024 • 2 new comments

• Add pad_seq_to_mult to sample_sequence_batch

  #10614 commented on Oct 2, 2024 • 2 new comments

• Add ModelOpt transformer model pruning example for Llama models, default to llama3.1-8b-base

  #10294 commented on Oct 4, 2024 • 1 new comment

• Efficient streaming decoding for RNN-T and TDT (support partial hypotheses)

  #9106 commented on Oct 2, 2024 • 1 new comment

• Support Torch FSDP2 from megatron core

  #10545 commented on Oct 4, 2024 • 0 new comments

• build(deps): bump vllm from 0.5.3.post1 to 0.5.5 in /requirements

  #10528 commented on Oct 4, 2024 • 0 new comments

• nemo-ux cudagraph

  #10527 commented on Oct 4, 2024 • 0 new comments

• [Nemo CICD] flaky test

  #10520 commented on Oct 3, 2024 • 0 new comments

• RNN-T confidence fix

  #10519 commented on Oct 5, 2024 • 0 new comments

• Update s3 checkpoint doc bucket path

  #10513 commented on Oct 4, 2024 • 0 new comments

• Add slice_by_offset and dry_run Support for Tar Dataset Creation; New Script for Partial Conversion

  #10511 commented on Sep 30, 2024 • 0 new comments

• add cudagraph docs

  #10500 commented on Oct 3, 2024 • 0 new comments

• Updating straggler detection args in NeMo 1.0

  #10493 commented on Oct 2, 2024 • 0 new comments

• Eval_beamsearch_ngram_ctc throws got an unexpected keyword argument 'logprobs'

  #10175 commented on Sep 29, 2024 • 0 new comments

• Make nemo text processing optional in TTS

  #10584 commented on Sep 30, 2024 • 0 new comments

• make /home/TestData readonly

  #10589 commented on Oct 4, 2024 • 0 new comments

• Change dist ckpt defaults

  #10590 commented on Oct 3, 2024 • 0 new comments

• Mistral-NeMo-12B recipe

  #10607 commented on Oct 3, 2024 • 0 new comments

• [NeMo-UX] Support `load_strictness`

  #10612 commented on Oct 2, 2024 • 0 new comments

• Add evaluate method and other minor fixes

  #10621 commented on Sep 30, 2024 • 0 new comments

• Use NCCL bootsrap backend for TP communication overlaps

  #10622 commented on Oct 1, 2024 • 0 new comments

• Add CI tests for SFT/PEFT

  #10632 commented on Oct 4, 2024 • 0 new comments

• fix: MegatronGPTModel get_forward_output_only_func position_ids=None

  #10653 commented on Oct 2, 2024 • 0 new comments

• ci: Switch to reusable workflows

  #10657 commented on Oct 4, 2024 • 0 new comments

• Check for meta tensors in checkpoint

  #10661 commented on Sep 30, 2024 • 0 new comments

• Respect load strictness when calling load_state_dict

  #10665 commented on Oct 2, 2024 • 0 new comments

• Add slimpajama example

  #10671 commented on Oct 4, 2024 • 0 new comments

• Converting HF model to Nemo gets an error

  #10264 commented on Sep 30, 2024 • 0 new comments

• Speaker Diarization Inference error with pickle

  #3421 commented on Oct 1, 2024 • 0 new comments

• Canary-1b on long audio file.

  #10487 commented on Oct 2, 2024 • 0 new comments

• [rank1]: AttributeError: 'NoneType' object has no attribute 'get' (finetuning Mamba Hybrid)

  #10285 commented on Oct 3, 2024 • 0 new comments

• ASR - WER not decreasing after certain point (Finetuning hybrid_cache_aware_streaming model)

  #10578 commented on Oct 3, 2024 • 0 new comments

• dim unmatch when doing sft with tensor parallel and sequence parallel and LoRA

  #10280 commented on Oct 4, 2024 • 0 new comments

• fastconformer hybrid recipe reports strange val_WER with `nemo:24.07` and `nemo:dev`

  #10299 commented on Oct 4, 2024 • 0 new comments

• Refactor: multi-node save best model only when needed

  #8277 commented on Oct 4, 2024 • 0 new comments

• fix: fix diarisation pickle error

  #8773 commented on Oct 2, 2024 • 0 new comments

• Fixed chokepoint in diarization for longer audios

  #9114 commented on Sep 29, 2024 • 0 new comments

• Making TDT models support all-positive durations (previously duration must contain 0)

  #9656 commented on Oct 6, 2024 • 0 new comments

• read system token from data in gpt_sft_chat_dataset

  #9775 commented on Oct 3, 2024 • 0 new comments

• check TB is enabled

  #10125 commented on Oct 1, 2024 • 0 new comments

• [Docs] Fix doc warnings, focus on feature and multimodal sections

  #10171 commented on Oct 1, 2024 • 0 new comments

• Akoumparouli/te lora gemm fork

  #10176 commented on Oct 4, 2024 • 0 new comments

• log step time at end

  #10202 commented on Oct 2, 2024 • 0 new comments

• McoreDistributedOptimizer wrapper

  #10231 commented on Sep 30, 2024 • 0 new comments

• Param and Grad Debug Logger

  #10236 commented on Oct 4, 2024 • 0 new comments

• Integrating mcore export

  #10238 commented on Oct 4, 2024 • 0 new comments

• Fix import path for CTCDecodingConfig

  #10286 commented on Oct 2, 2024 • 0 new comments

• Add Tiktoken support for TRTLLM

  #10306 commented on Oct 2, 2024 • 0 new comments

• TE: if cuda is not available raise ImportError

  #10340 commented on Sep 30, 2024 • 0 new comments

• Support custom lightning profilers in config

  #10364 commented on Oct 4, 2024 • 0 new comments

• Fix trascribe speech parralel with tarred datasets

  #10372 commented on Oct 6, 2024 • 0 new comments

• Reflect CLI change nemorun -> nemo

  #10443 commented on Oct 2, 2024 • 0 new comments

• MoE Upcycling support for GPT based models

  #10485 commented on Sep 30, 2024 • 0 new comments