SupraLabs 发布 5M 参数 GatedDeltaNet 模型 SupraGDN-5M

SupraLabs SupraLabs/SupraGDN-5M SupraLabs/SupraGDN-5M https://huggingface.co/SupraLabs/SupraGDN-5M

SupraLabs/SupraGDN-5M · Hugging Face Hugging Face (https://huggingface.co/) Models (https://huggingface.co/models) Datasets (https://huggingface.co/datasets) Spaces (https://huggingface.co/spaces) Buckets new (https://huggingface.co/storage) Docs (https://huggingface.co/docs) Enterprise (https://huggingface.co/enterprise) Pricing (https://huggingface.co/pricing) Website Tasks (https://huggingface.co/tasks) HuggingChat (https://huggingface.co/chat) Collections (https://huggingface.co/collections) Languages (https://huggingface.co/languages) Organizations (https://huggingface.co/organizations) Community Blog (https://huggingface.co/blog) Posts (https://huggingface.co/posts) Daily Papers (https://huggingface.co/papers) Hardware (https://huggingface.co/hardware) Learn (https://huggingface.co/learn) Discord (https://huggingface.co/join/discord) Forum (https://discuss.huggingface.co/) GitHub (https://github.com/huggingface) Solutions Team & Enterprise (https://huggingface.co/enterprise) Hugging Face PRO (https://huggingface.co/pro) Enterprise Support (https://huggingface.co/support) Inference Providers (https://huggingface.co/inference/models) Inference Endpoints (https://huggingface.co/inference-endpoints) Storage Buckets (https://huggingface.co/storage) Log In (https://huggingface.co/login) Sign Up (https://huggingface.co/join) (https://huggingface.co/SupraLabs) SupraLabs (https://huggingface.co/SupraLabs) / SupraGDN-5M (https://huggingface.co/SupraLabs/SupraGDN-5M) like 6 Follow SupraLabs 389 License: apache-2.0 Model card (https://huggingface.co/SupraLabs/SupraGDN-5M) Files Files and versions xet (https://huggingface.co/SupraLabs/SupraGDN-5M/tree/main) Community (https://huggingface.co/SupraLabs/SupraGDN-5M/discussions) Copy to bucket new Pretraining (https://huggingface.co/SupraLabs/SupraGDN-5M#pretraining) Final Loss (https://huggingface.co/SupraLabs/SupraGDN-5M#final-loss) Samples (https://huggingface.co/SupraLabs/SupraGDN-5M#samples) Benchmarks (https://huggingface.co/SupraLabs/SupraGDN-5M#benchmarks) How this model compares to other SOTA mini ~5M parameter models (https://huggingface.co/SupraLabs/SupraGDN-5M#how-this-model-compares-to-other-sota-mini-5m-parameter-models) How to run the model (https://huggingface.co/SupraLabs/SupraGDN-5M#how-to-run-the-model) SupraGDN-5M • GatedDeltaNet • Tiny-SOTA (https://cdn-uploads.huggingface.co/production/uploads/697f2832c2c5e4daa93cece7/vTXgrE4X7D0o4NLdkddVX.png) We are introducing SupraGDN-5M, a GatedDeltaNet-architecture-based model with 5 million parameters, pretrained from scratch as a base model on a single RTX Pro 4500 SE on 5B Fineweb-Edu tokens in about ~2.5 hours. This is the base work for future models, as the upcoming Supra3-family with our most capable tiny SOTA SLM models. Please note, that this is an undertrained, experimental base model with no high capabilities. (https://huggingface.co/SupraLabs/SupraGDN-5M#pretraining) Pretraining The pretraining ran for exactly 1 epoch on the first 5B tokens of Fineweb-Edu sample-10BT on a single RTX Pro 4500 SE rented from Runpod. Batch Size: 1024 Context: 256 tokens Pretraining tokens: 5B Vocab size of custom tokenizer: 6000 tokens GPU: RTX Pro 4500 SE on Runpod Total time: ~2.5 hours Total cost: ~$2 (https://huggingface.co/SupraLabs/SupraGDN-5M#final-loss) Final Loss The Val Loss dropped from ~7.1 to ~3.5251 and a Val-PPL of 33.96. Same goes for the Train Loss. (https://huggingface.co/SupraLabs/SupraGDN-5M#samples) Samples PROMPT: The history of OUTPUT: The history of Native Americans during the Civil War was much worse during the Civil War. Since then, and the culture of North America has changed, the history of Native Americans from the beginning, the story of the revolutionary colonies of America, and the history of America has changed dramatically. The history of North America in 1876 marks the twentieth-morning of America’ ----------------------------------------------------