FPGA based Vision Transformer accelerator (Harvard CS205)

• cd code/code_cpu
• Everything below is done in the code/code_cpu directory!!!

• wget https://download.pytorch.org/libtorch/nightly/cpu/libtorch-shared-with-deps-latest.zip
• unzip libtorch-shared-with-deps-latest.zip

• mkdir build
• cd build
• cmake -DCMAKE_PREFIX_PATH=/absolute/path/to/libtorch_folder/that/you/just/downloaded ..
• cmake --build . --config Release
• cd .. (we exit the build directory)

• python download_mnist.py
• mkdir mnist
• Put all the downloaded .ubyte files into ./mnist folder

• cd build
• run ./main with either test (test model with loaded weights) or train
• You can move job.sh into build directory and run ./job.sh to get time benchmarks in output.txt
• Pretrained weights can be downloaded here: Drive (90% acc) or use the one in the folder (~86% accuracy)

Please checkout code/fpga/test_accel/test_accel.srcs/sim_1/imports/src/ for the testbenches. Please also import the files in code/fpga/test_accel/test_accel.srcs/sources_1/imports/src/ as the rtl sources. The testbench demonstrates the controller working in parallel with the DMA engine "spad_arbiter". Please feel free to email Eric for further questions.