Alex Karavaev

💎 𝗔 𝘃𝗶𝘀𝘂𝗮𝗹 𝗴𝘂𝗶𝗱𝗲 𝘁𝗼 𝗠𝗶𝘅𝘁𝘂𝗿𝗲 𝗼𝗳 𝗘𝘅𝗽𝗲𝗿𝘁𝘀 by Maarten Grootendorst ● 𝗠𝗼𝗘 aims to improve the efficiency of traditional dense layers (FFNNs) in LLMs, where all parameters are activated regardless of the input. In contrast, MoE employs sparse layers where only a subset of experts are activated based on the given input. ● Although a given MoE has more parameters to load (𝘀𝗽𝗮𝗿𝘀𝗲 𝗽𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿𝘀), fewer are activated since we only use some experts during inference (𝗮𝗰𝘁𝗶𝘃𝗲 𝗽𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿𝘀). MoE models need more VRAM to load in all experts but run faster during inference. ● 𝗠𝗶𝘅𝘁𝗿𝗮𝗹 8𝘅7𝗕:  We have to load 8 experts but we only need to use 2 experts for inference. ● 𝗘𝘅𝗽𝗲𝗿𝘁𝘀 are specialized in specific domains like "Psychology" but rather in handling specific tokens in specific contexts. ● 𝗧𝗵𝗲 𝗿𝗼𝘂𝘁𝗲𝗿 is the critical component of an MoE system, as it determines which experts are selected for processing a given input token. This selection process is important during the training and inference phases. ● A simple router implementation, often leads to imbalanced expert selection, hindering both training and inference performance. 𝗟𝗼𝗮𝗱 𝗯𝗮𝗹𝗮𝗻𝗰𝗶𝗻𝗴 techniques are employed to address the expert selection bias aiming to distribute the workload evenly among experts. ● 𝗞𝗲𝗲𝗽𝗧𝗼𝗽𝗞: After adding noise, only the top "k" experts with the highest probabilities are selected for processing the input tokens. ● 𝗟𝗼𝗮𝗱 𝗯𝗮𝗹𝗮𝗻𝗰𝗶𝗻𝗴 𝗹𝗼𝘀𝘀, incorporated into the training process to encourage expert balance, operates by penalizing uneven expert selection, promoting a more uniform distribution of workload among experts. ● 𝗦𝗶𝗺𝗽𝗹𝗶𝗳𝘆𝗶𝗻𝗴 𝗠𝗼𝗘 𝘄𝗶𝘁𝗵 𝘁𝗵𝗲 𝗦𝘄𝗶𝘁𝗰𝗵 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿: Switch Transformer is a T5 model (encoder-decoder) that replaces the traditional FFNN layer with a Switching Layer. The Switching Layer is a Sparse MoE layer that selects a single expert for each token (Top-1 routing). ● 𝗩𝗶𝘀𝗶𝗼𝗻-𝗠𝗼𝗘 (𝗩-𝗠𝗼𝗘) is one of the first implementations of MoE in an image model.7 It takes the ViT as we saw before and replaces the dense FFNN in the encoder with a Sparse MoE. 👉 https://lnkd.in/dwY6BE3N ♻️𝒌𝒊𝒏𝒅𝒍𝒚 𝒓𝒆𝒑𝒐𝒔𝒕♻️ #moe #

20

This how you approximately calculate GPU memory required for any LLM. Read “How Much GPU Memory is Needed to Serve a Large Language Model (LLM)?“ by Mastering LLM (Large Language Model) on Medium: https://lnkd.in/dh9V5-8s

6

Facebook just released a very different kind of LLMs. These are focused on code optimization and compiler optimization. This is super important because it can fundamentally change the way code is executed on any machine. Here's how it works right now: Programs written in many languages like C/C have intermediate steps before they are run on the computer. For example they are converted into Byte Code and then converted into machine code via LLVM (Low level virtual machine). But Facebook's new model can take in the byte code as input and convert it to into a more efficient representation so that when this new code is converted into machine code, it will be faster. "In theory", it could mean that all existing code might be made faster without any change by developers. Our games can have higher frame rates on the same GPUs. Photoshop might run faster on the same laptop. Our software might complete tasks faster without any changes at all. But that will probably take quite a long time. To be honest I don't expect these models to actually be used in production since unless they were 100% as good as compilers they will probably break the code. And are they 100% good? No, not even close. But the speed at which improvements are happening, they will probably reach that level eventually. And that's when we will see models like these everywhere. 

11