While nanoGPT is a great project and an excellent resource; it is, by design, very minimal² and limited in its flexibility.

Working through the code I found myself making minor changes here and there to test new ideas and run variations on different experiments. These changes eventually built to the point where my {goals, scope, code} for the project had diverged significantly from the original vision.

As a result, I figured it made more sense to move things to a new project, wordplay.

I’ve priortized adding functionality that I have found to be useful or interesting, but am absolutely open to input or suggestions for improvement.

Different aspects of this project have been motivated by some of my recent work on LLMs.

• Projects:
  • ezpz: Painless distributed training with your favorite {framework, backend} combo.
  • Megatron-DeepSpeed: Ongoing research training transformer language models at scale, including: BERT & GPT-2
• Collaboration(s):
  • DeepSpeed4Science (2023-09)
    • Loooooooong Sequence Lengths
    • Project Website
    • Preprint Song et al. (2023)
    • Blog Post
    • Tutorial
  • GenSLMs:
    • GitHub
    • Preprint
    • 🏆 ACM Gordon Bell Special Prize for COVID-19 Research
• Talks / Workshops:
  • LLM-lunch-talk (2023-10-12): LLMs at ALCF.
    • Slides
    • GitHub
  • Creating Small(-ish) LLMs (2023-11-30)
    • Workshop
    • Slides
    • GitHub

The easiest way to get the most recent version is to:

python3 -m pip install "git https://github.com/saforem2/wordplay.git"

If you’d like to work with the project and run / change things yourself, I’d recommend installing from a local (editable) clone of this repository:

git clone "https://github.com/saforem2/wordplay"
cd wordplay
mkdir v venv
python3 -m venv venv --system-site-packages
source venv/bin/activate
python3 -m pip install -e .