Abhinav Kohar

I recently read “Towards Monosemanticity: Decomposing Language Models with Dictionary Learning” by Bricken et al 2023. Published at transformer-circuits.pub In this paper the authors discuss the use of autoencoders to extract interpretable features from a one-layer transformer model. The approach aims to improve the understanding and transparency of artificial neural networks. To do this they use the concept of dictionary learning which is a method used to decompose the activations of the transformer model into a set of basis features that are sparse and interpretable. This involves training sparse autoencoders to identify a dictionary of monosemantic features (or features with a single meaning). These monosemantic features in single units (artificial neurons) reminds me of the “grandmother neuron” theory in neuroscience which proposes that specific neurons respond to specific highly complex stimuli. However, in real brains, it is widely believed that complex representations are encoded by the collective activity of groups of neurons (and not monosemantic). But going back to the paper, I really like the visualization tools and illustration to make such complex networks more transparent (as opposed to black boxes). Paper: https://lnkd.in/gR87pJcs

3

AI: the bubble that burst What’s wrong with the current unscientific AI. Let’s summarize the WSJ article: * Uncertain return to investors * Unrealistic goals * Not delivering on its promises * Narrow set of practical applications * Many model variations of the same algorithm * Taking a long time, or never, to realize the business case * Spending does not guarantee a return * Pace of improvement slowing, or too many errors * Imperfect data; data being appropriated from others; * Open source options catching up with paywall models * Excessively expensive to run and maintain for meager results * Public starting to realize of its limits: self-driving cars fatal accidents; poor search results; lawsuits by owners of the data; driven by personalities not science; increasingly taking more electrical power from the grid Finally, at the end of the article, WSJ closes with: “The problem is that the current level of investment—in startups and by big companies—seems to be predicated on the idea that AI is going to get so much better, so fast, and be adopted so quickly that its impact on our lives and the economy is hard to comprehend. Mounting evidence suggests that won’t be the case.” Source: The Wall Street Journal https://lnkd.in/gg-7hJjN #artificialIntelligence #ai #machineLearning #dataScience #spe #petroleumEngineering

155

61 Comments

The future of QUANTUM machine learning could be closer than previously envisaged. Thanks to the on-going research and development efforts of the likes of Google, IBM, Cambridge, etc. For those handling big data and complex algorithms such as deep learning, machine learning is beginning to reach the limit of the traditional computing paradigm. Imagine training a deep learning model for days, weeks, and months. The training time of GPT-4 is around five to six months. So, imagine the huge energy consumption. Quantum computer, and quantum machine learning in particular, is poised to significantly reduce the training time to seconds or minutes as well as the associated energy consumption. Be ready! This article gives a succinct introduction. https://lnkd.in/dYfwX5Rb #quantumcomputing #quantummachinelearning #quantumml #machinelearning #ml #deeplearning #quantumcomputer #chatgpt4

24

6 Comments

This approach could turn the Generative AI industry upside down. Most everyone knows that AI requires computers to do a lot of calculations and especially for Large Language Models (LLMs) like ChatGPT. The majority of these calculations are actually “Matrix Multiplications”. (You remember, a 3x3 matrix times another 3x3 matrix). If the researchers in the article are proven correct, they have demonstrated a way that Matrix Multiplication can be replaced with a more efficient alternative inside the LLM getting nearly the same results. The article positions this savings as a win for the environment (yeah environment), but I think they are missing the boat. More efficient means less electricity and potentially a smaller need for GPU resources, the bread and butter of stock darling Nvidia. When you combine this cost saving with reduced privacy concerns of hosted LLMs (Open AI) and innovative LLM licensing schemes (LibreChat - https://lnkd.in/gKQjMfz5) you can see the cracks in the current direction of the LLM industry. Q: If you could run your own LLM on your own private environment with MIT open-source licensing and without expensive GPUs, why exactly do you need Nvidia, Open AI and alike? Bold prediction: if the researchers get this working at scale, we could see a rapid shift and commoditization in the LLM and generative AI space. All the pieces are in place. TLDR details from the article, describing the innovation: “The researchers' approach involves two main innovations: first, they created a custom LLM and constrained it to use only ternary values (-1, 0, 1) instead of traditional floating-point numbers, which allows for simpler computations. Second, the researchers redesigned the computationally expensive self-attention mechanism in traditional language models with a simpler, more efficient unit (that they called a MatMul-free Linear Gated Recurrent Unit—or MLGRU) that processes words sequentially using basic arithmetic operations instead of matrix multiplications.” #AI #GenerativeAI #Innovation

24

It was a superb chatting with Andrew Beam and Arjun Manrai. Thank you both for the long form conversation! Refreshing break from the "bite sized" world enabled by NEJM AI. What we talked about was possible given the stellar research team I get to work with (Jason Fries, Alison Callahan, Michael Wornow, Suhana Bedi, Akshay Swaminathan, Zepeng Frazier Huo, Scott Fleming and Ethan Steinberg) and the vibrant community at Stanford University School of Medicine, Stanford Health Care supported by the brilliant team at Technology & Digital Solutions - Stanford Medicine. Some of the ideas we discussed are also outlined in blogs at our Stanford Institute for Human-Centered Artificial Intelligence (HAI). e.g. 1. How can foundation models advance healthcare - https://lnkd.in/eiBw4YHr 2. How do we ensure healthcare ai is useful - https://lnkd.in/eDTc7Dqi 3. Are we there yet with LLMs in healthcare? - https://lnkd.in/eDQv2zH4 #HealthcareAI

277

4 Comments

Conditional Computation in LLMs (MoEs): Understanding the Concept of MoEs The Mixture of Experts (MoE) works on the idea of conditional computation, allowing the model to only run through some parts (an Expert) for a given input compared to a dense model where all parameters are used. It consists of a gating mechanism that selects the experts for the given input, enabling the model to scale without increasing the required computation. Popular LLM architectures like Mixtral and Grok use the same concept. What is MoEs? The Mixture of Experts (MoE) makes a simple modification to the usual architecture by replacing all or some of the Feed Forward layers with an MoE layer. The MoE layer consists of: 1. Router: A parameter gating mechanism that selects the experts used to process the given input by producing a probability distribution over experts. 2. Experts: Standalone NN modules with independent sets of parameters trained jointly with routers. It can use more complex architecture to create hierarchical MoE modules by implementing each expert as another MoE. - Implementation in Switch Transformers : Training was found to be 7x faster than training the base T5, keeping computation and training time constant. - Gshard and MeshTransformer : These frameworks propose efficient methods to train the MoE architectures using the parallelisation of models across GPUs/TPUs. Challenges with MoEs and Solutions 1. Inconsistent Batch Size for Experts: Can be overcome with the use of "soft" constraints in training loss. 2. Load Balancing Across Experts: The issue of repeatedly using the same few experts while training can be overcome by including better balancing mechanisms. 3. Model Instability Issues: Use of router-z Loss as an auxiliary loss to encourage smaller values within the router mechanism. 4. Model Overfitting: Requires different hyperparameter settings for effective fine-tuning. It was found that models benefit from high learning rates and smaller batch sizes. Recommended Reading/References 1. Hugging Face Blog on MoE - https://lnkd.in/g36gQJTx 2. Cameron Wolfe's Substack on Conditional Computation - https://lnkd.in/gtpjMHnJ

60

1 Comment

While Linkedin Space is overwhelmed with GenAI contents , there has been an excellent drive going on by Scientific community on how AI (Neural Net and other DL/ML s) can help solving science probems of solving equations and simulations. It opened my eyes on vast opportinuty of possibilities. For further, please read Physics Inspired Neural Net, Neural Operator ... More here .. https://lnkd.in/eNRtswmr (X page of Anima Anandkumar ) https://sciml.ai/ #sciml #aiforscience

18

1 Comment