Mini-gpts: Efficient large language models through contextual pruning
T Valicenti, J Vidal, R Patnaik - arXiv preprint arXiv:2312.12682, 2023 - arxiv.org
T Valicenti, J Vidal, R Patnaik
arXiv preprint arXiv:2312.12682, 2023•arxiv.orgIn AI research, the optimization of Large Language Models (LLMs) remains a significant
challenge, crucial for advancing the field's practical applications and sustainability. Building
upon the foundational work of Professor Song Han's lab at MIT, this paper introduces a
novel approach in developing Mini-GPTs via contextual pruning. Our methodology
strategically prunes the computational architecture of traditional LLMs, like Phi-1.5, focusing
on retaining core functionalities while drastically reducing model sizes. We employ the …
challenge, crucial for advancing the field's practical applications and sustainability. Building
upon the foundational work of Professor Song Han's lab at MIT, this paper introduces a
novel approach in developing Mini-GPTs via contextual pruning. Our methodology
strategically prunes the computational architecture of traditional LLMs, like Phi-1.5, focusing
on retaining core functionalities while drastically reducing model sizes. We employ the …
In AI research, the optimization of Large Language Models (LLMs) remains a significant challenge, crucial for advancing the field's practical applications and sustainability. Building upon the foundational work of Professor Song Han's lab at MIT, this paper introduces a novel approach in developing Mini-GPTs via contextual pruning. Our methodology strategically prunes the computational architecture of traditional LLMs, like Phi-1.5, focusing on retaining core functionalities while drastically reducing model sizes. We employ the technique across diverse and complex datasets, including US law, Medical Q&A, Skyrim dialogue, English-Taiwanese translation, and Economics articles. The results underscore the efficiency and effectiveness of contextual pruning, not merely as a theoretical concept but as a practical tool in developing domain-specific, resource-efficient LLMs. Contextual pruning is a promising method for building domain-specific LLMs, and this research is a building block towards future development with more hardware compute, refined fine-tuning, and quantization.
arxiv.org
Показан е най-добрият резултат за това търсене. Показване на всички резултати