Better & Faster Large Language Models via Multi-token Prediction - A novel training method for large language models that improves efficiency and performance.

## Innovation of Multi-token Prediction The multi-token prediction method innovates by predicting multiple future tokens at each position in the training corpus, rather than just the next token. Each token's loss is calculated independently, and the method optimizes the order of forward and backward propagation to reduce GPU memory usage without increasing training time. ## Performance Improvement in LLMs Multi-token prediction improves LLM performance by increasing sample efficiency and reducing GPU memory usage. It achieves significant performance gains in coding and natural language tasks, with up to 3x faster inference speeds. For example, a 13B parameter model solved 12% more problems on HumanEval and 17% more on MBPP. ## Key Features of Multi-token Prediction The key features of the multi-token prediction method include: - Simultaneous prediction of multiple future tokens. - Reduced GPU memory usage through optimized propagation order. - Improved sample efficiency, especially in larger models. - Enhanced performance in coding and natural language tasks. - Up to 3x faster inference speeds. ## Performance Improvement in LLMs The potential applications of multi-token prediction include: - Training LLMs for higher efficiency and performance, particularly in generative tasks like coding. - Real-time applications due to faster inference speeds, such as speculative decoding for 2.7x to 3x acceleration. ## Evaluation Datasets The multi-token prediction method was evaluated using datasets such as MBPP, HumanEval, CodeContests, and GSM8K. These datasets were used to measure performance improvements in coding and natural language tasks. ## Environmental Impact Training models with multi-token prediction requires approximately 500,000 GPU hours (A100-80GB, H100), estimated to emit about 50 tons of CO2eq. However, these emissions are fully offset by Meta's sustainability initiatives. ## Future Research Directions Future research directions for multi-token prediction include exploring its application in larger natural language models, optimizing configurations for further performance improvements, and investigating its advantages in smaller models and algorithmic reasoning tasks. ### Citation sources: - [Better & Faster Large Language Models via Multi-token Prediction](https://arXiv.org/pdf/2404.19737) - Official URL Updated: 2025-03-28