Llion Jones

Graph Machine Learning in the Era of Large Language Models (Hong Kong Polytechnic University, April 2024) Paper: https://lnkd.in/gTxVyM_4 Abstract: "Graphs play an important role in representing complex relationships in various domains like social networks, knowledge graphs, and molecular discovery. With the advent of deep learning, Graph Neural Networks (GNNs) have emerged as a cornerstone in Graph Machine Learning (Graph ML), facilitating the representation and processing of graph structures. Recently, LLMs have demonstrated unprecedented capabilities in language tasks and are widely adopted in a variety of applications such as computer vision and recommender systems. This remarkable success has also attracted interest in applying LLMs to the graph domain. Increasing efforts have been made to explore the potential of LLMs in advancing Graph ML's generalization, transferability, and few-shot learning ability. Meanwhile, graphs, especially knowledge graphs, are rich in reliable factual knowledge, which can be utilized to enhance the reasoning capabilities of LLMs and potentially alleviate their limitations such as hallucinations and the lack of explainability. Given the rapid progress of this research direction, a systematic review summarizing the latest advancements for Graph ML in the era of LLMs is necessary to provide an in-depth understanding to researchers and practitioners. Therefore, in this survey, we first review the recent developments in Graph ML. We then explore how LLMs can be utilized to enhance the quality of graph features, alleviate the reliance on labeled data, and address challenges such as graph heterogeneity and out-of-distribution (OOD) generalization. Afterward, we delve into how graphs can enhance LLMs, highlighting their abilities to enhance LLM pre-training and inference. Furthermore, we investigate various applications and discuss the potential future directions in this promising field."

6

Unveiling Schrödinger’s Memory: Dynamic Memory Mechanisms in Transformer-Based Language Models Hong Kong Polytechnic University researchers use the Universal Approximation Theorem (UAT) to explain memory in LLMs. They propose that LLM memory, termed “Schrödinger’s memory,” is only observable when queried, as its presence remains indeterminate otherwise. Using UAT, they argue that LLMs dynamically approximate past information based on input cues, resembling memory. Their study introduces a new method to assess LLM memory abilities and compares LLMs’ memory and reasoning capacities to those of humans, highlighting both similarities and differences. The study also provides theoretical and experimental evidence supporting LLMs’ memory capabilities.... Read full article here: https://lnkd.in/g3MhFUAX Paper: https://lnkd.in/gU-qt2Dt

22

This week in #TheBatch: 🎨 All about Claude’s Artifacts feature 🌱 AI growth hinders carbon emissions goals 🧠 GaLore, a memory-efficient method for fine-tuning and pretraining Plus: Andrew Ng shares his concerns about the proposed California regulation SB 1047. Read The Batch now: https://hubs.la/Q02G5YpM0

49

Huge release by Daekeun Kim and Hyo (HK) Choi from our AI GBB team in Korea, sharing Korean language benchmarks for our latest Azure OpenAI model, gpt-4o-mini - like gpt-4o, the model excels on Korean language tasks, in this case mini completely eclipses gpt-35-turbo and nearly achieves gpt-4-turbo performance. The benchmarks and code are open source, extensible beyond our Azure models, so we encourage our customers and the larger community to try it out and even fork/share your own findings - and please let me in the comments what other Asia languages beyond Korean you may be looking to evaluate with your language models.

52

7件のコメント

Releasing a float32 tensor compression pip package tonight: https://lnkd.in/gDTAmkrq This achieves a 4:1 compression ratio for pre-trained models with a guaranteed precision of 1e-4 max error in the values. It [de]compresses at about 1.5GBPS by using CUDA for acceleration. pip install -U cuda_float_compress

10

Aligning Large Language Models with Diverse User Preferences Using Multifaceted System Messages: The JANUS Approach Quick read: https://lnkd.in/gQAsT95M Paper: https://lnkd.in/gkBAuWiC

19

Facebook just released a very different kind of LLMs. These are focused on code optimization and compiler optimization. This is super important because it can fundamentally change the way code is executed on any machine. Here's how it works right now: Programs written in many languages like C/C have intermediate steps before they are run on the computer. For example they are converted into Byte Code and then converted into machine code via LLVM (Low level virtual machine). But Facebook's new model can take in the byte code as input and convert it to into a more efficient representation so that when this new code is converted into machine code, it will be faster. "In theory", it could mean that all existing code might be made faster without any change by developers. Our games can have higher frame rates on the same GPUs. Photoshop might run faster on the same laptop. Our software might complete tasks faster without any changes at all. But that will probably take quite a long time. To be honest I don't expect these models to actually be used in production since unless they were 100% as good as compilers they will probably break the code. And are they 100% good? No, not even close. But the speed at which improvements are happening, they will probably reach that level eventually. And that's when we will see models like these everywhere. 

11

Efficient Large Language Model Training: GaLore and Q-Galore Revolutionize Memory Management Training and fine-tuning large language models (LLMs) demand substantial computational resources, especially in terms of memory management. Traditional training methods require significant memory to manage numerous parameters and optimization states. For instance, training the Llama 7B model requires around 58 GB of memory, broken down as follows: 14 GB for trainable parameters 42 GB for Adam optimizer states and weight gradients 2 GB for activations Several techniques have been developed to address the challenge of high memory usage, including designing smaller scale LLMs and employing methods like LoRA (Low-Rank Adaptation) to optimize weights. A promising approach, Gradient Low-Rank Projection (GaLore), focuses on optimizing gradient computations during model fine-tuning. GaLore is a memory-efficient low-rank training strategy that enables full-parameter learning while being more memory-efficient than traditional low-rank adaptation methods such as LoRA. By projecting gradients into a lower-dimensional space, GaLore reduces the computational overhead associated with weight updates in large models, making fine-tuning faster and more resource-efficient. This approach is particularly beneficial for training large models on hardware with limited computational capacity. GaLore has demonstrated a significant reduction in memory usage—by up to 63.3%—enabling the training of the Llama 7B model with just 24 GB of memory. This method achieves full parameter training for LLMs through low-rank gradient updates using Singular Value Decomposition (SVD). Building on the GaLore approach, Q-Galore takes the optimization a step further by converting floating-point numbers to 8-bit integers. This quantization process speeds up computations and further reduces memory usage, enhancing the efficiency of training large language models. Q-Galore represents an innovative advancement in low-rank gradient updates, making it possible to train and fine-tune substantial models with significantly reduced resource requirements. For comments reach out to [email protected] Innovation Hacks AI #LLM #AI #FineTuning #PromptEngineering #MachineLearning #Innovation

2