rmsnorm
Here are 4 public repositories matching this topic...
Efficient kernel for RMS normalization with fused operations, includes both forward and backward passes, compatibility with PyTorch.
-
Updated
Jun 5, 2024 - Python
Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps. Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE). Tested on Taylor Swift song lyrics dataset.
-
Updated
Nov 18, 2024 - Python
Improve this page
Add a description, image, and links to the rmsnorm topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the rmsnorm topic, visit your repo's landing page and select "manage topics."