Apple Introduce IDEA Prune Pipeline per l'Addestramento Efficiente di Modelli Linguistici Generativi
· By Antonio Sedino, CTRO · Published by Reinventy Solutions Corp.
I pipeline di potatura strutturata mostrano promesse di efficienza dei token rispetto all'addestramento di modelli target-size da zero, sostenendo l'incorporamento di un pre-addestramento del modello ingrandito.
Recent advancements in large language models have intensified the need for efficient and deployable models within limited inference budgets. Structured pruning pipelines have shown promise in token efficiency compared to training target-size models from scratch. In this paper, we advocate incorporating enlarged model pretraining to improve efficiency. The IDEA Prune pipeline integrates enlargement and pruning stages to achieve better token efficiency in generative language model pretraining.
Read the original source at Apple Machine Learning Research ↗