Librarian Bot. I found the following papers similar to this paper. \n
The following papers were recommended by the Semantic Scholar API
\n- \n
- AdaLomo: Low-memory Optimization with Adaptive Learning Rate (2023) \n
- Training and inference of large language models using 8-bit floating point (2023) \n
- QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources (2023) \n
- BitNet: Scaling 1-bit Transformers for Large Language Models (2023) \n
- Baichuan 2: Open Large-scale Language Models (2023) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n","updatedAt":"2023-11-03T16:44:43.050Z","author":{"avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isMod":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.75470370054245},"editors":["librarian-bot"],"reactions":[{"reaction":"👍","users":["neuralink"],"count":1}],"isReport":false}},{"id":"6665391a9b7d75d2edfa359b","author":{"avatarUrl":"/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg","fullname":"Julien BLANCHON","name":"blanchon","type":"user","isPro":true,"isHf":false,"isMod":false},"createdAt":"2024-06-09T05:09:46.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"# Unleashing FP8 Power: Efficiently Training Massive LLMs\n\nhttps://cdn-uploads.huggingface.co/production/uploads/6186ddf6a7717cb375090c01/oX3hPRGxH62ITBKzrgoQf.mp4 \n\n## Links 🔗:\n👉 Subscribe: https://www.youtube.com/@Arxflix\n👉 Twitter: https://x.com/arxflix\n👉 LMNT (Partner): https://lmnt.com/\n\n\nBy Arxflix\n","html":"Unleashing FP8 Power: Efficiently Training Massive LLMs
\n \n\nLinks 🔗:
\n👉 Subscribe: https://www.youtube.com/@Arxflix
👉 Twitter: https://x.com/arxflix
👉 LMNT (Partner): https://lmnt.com/
By Arxflix
Papers
arxiv:2310.18313
FP8-LM: Training FP8 Large Language Models
Abstract
In this paper, we explore FP8 low-bit data formats for efficient training of large language models (LLMs). Our key insight is that most variables, such as gradients and optimizer states, in LLM training can employ low-precision data formats without compromising model accuracy and requiring no changes to hyper-parameters. Specifically, we propose a new FP8 automatic mixed-precision framework for training LLMs. This framework offers three levels of FP8 utilization to streamline mixed-precision and distributed parallel training for LLMs. It gradually incorporates 8-bit gradients, optimizer states, and distributed learning in an incremental manner. Experiment results show that, during the training of GPT-175B model on H100 GPU platform, our FP8 mixed-precision training framework not only achieved a remarkable 42% reduction in real memory usage but also ran 64% faster than the widely adopted BF16 framework (i.e., Megatron-LM), surpassing the speed of Nvidia Transformer Engine by 17%. This largely reduces the training costs for large foundation models. Furthermore, our FP8 mixed-precision training methodology is generic. It can be seamlessly applied to other tasks such as LLM instruction tuning and reinforcement learning with human feedback, offering savings in fine-tuning expenses. Our FP8 low-precision training framework is open-sourced at {https://github.com/Azure/MS-AMP}{aka.ms/MS.AMP}.
Community
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- AdaLomo: Low-memory Optimization with Adaptive Learning Rate (2023)
- Training and inference of large language models using 8-bit floating point (2023)
- QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources (2023)
- BitNet: Scaling 1-bit Transformers for Large Language Models (2023)
- Baichuan 2: Open Large-scale Language Models (2023)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
Unleashing FP8 Power: Efficiently Training Massive LLMs
Links 🔗:
👉 Subscribe: https://www.youtube.com/@Arxflix
👉 Twitter: https://x.com/arxflix
👉 LMNT (Partner): https://lmnt.com/
By Arxflix
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2310.18313 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2310.18313 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2310.18313 in a Space README.md to link it from this page.