OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640开源了新模型

OpenMOSS OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640 OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640 https://huggingface.co/OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640

OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640 · Hugging Face Hugging Face (https://huggingface.co/) Models (https://huggingface.co/models) Datasets (https://huggingface.co/datasets) Spaces (https://huggingface.co/spaces) Buckets new (https://huggingface.co/storage) Docs (https://huggingface.co/docs) Enterprise (https://huggingface.co/enterprise) Pricing (https://huggingface.co/pricing) Website Tasks (https://huggingface.co/tasks) HuggingChat (https://huggingface.co/chat) Collections (https://huggingface.co/collections) Languages (https://huggingface.co/languages) Organizations (https://huggingface.co/organizations) Community Blog (https://huggingface.co/blog) Posts (https://huggingface.co/posts) Daily Papers (https://huggingface.co/papers) Hardware (https://huggingface.co/hardware) Learn (https://huggingface.co/learn) Discord (https://huggingface.co/join/discord) Forum (https://discuss.huggingface.co/) GitHub (https://github.com/huggingface) Solutions Team & Enterprise (https://huggingface.co/enterprise) Hugging Face PRO (https://huggingface.co/pro) Enterprise Support (https://huggingface.co/support) Inference Providers (https://huggingface.co/inference/models) Inference Endpoints (https://huggingface.co/inference-endpoints) Storage Buckets (https://huggingface.co/storage) Log In (https://huggingface.co/login) Sign Up (https://huggingface.co/join) (https://huggingface.co/OpenMOSS-Team) OpenMOSS-Team (https://huggingface.co/OpenMOSS-Team) / SmolLM-1.7B-base-mla-topk2-rank640 (https://huggingface.co/OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640) like 0 Follow OpenMOSS 1.18k Safetensors (https://huggingface.co/models?library=safetensors) llama (https://huggingface.co/models?other=llama) smollm (https://huggingface.co/models?other=smollm) mla (https://huggingface.co/models?other=mla) mlafication (https://huggingface.co/models?other=mlafication) delta-weights (https://huggingface.co/models?other=delta-weights) License: apache-2.0 Model card (https://huggingface.co/OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640) Files Files and versions xet (https://huggingface.co/OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640/tree/main) Community (https://huggingface.co/OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640/discussions) Copy to bucket new SmolLM-1.7B-base-mla-topk2-rank640 (https://huggingface.co/OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640#smollm-17b-base-mla-topk2-rank640) Variant (https://huggingface.co/OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640#variant) Loading (https://huggingface.co/OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640#loading) Files (https://huggingface.co/OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640#files) License (https://huggingface.co/OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640#license) (https://huggingface.co/OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640#smollm-17b-base-mla-topk2-rank640) SmolLM-1.7B-base-mla-topk2-rank640 This repository contains the attention-only incremental weights for an MLAfication conversion of HuggingFaceTB/SmolLM-1.7B (https://huggingface.co/HuggingFaceTB/SmolLM-1.7B) . It is not a standalone full-model checkpoint. (https://huggingface.co/OpenMOSS-Team/SmolLM-1.7B-base-mla-topk2-rank640#variant) Variant Field Value Base model HuggingFaceTB/SmolLM-1.7B Base revision d7449ff7241c863f3e8accc475155f0f97afa011 MLA latent rank 640 RoPE dimensions per KV head 2 Training checkpoint step 5600 Training variant stage1+2-distill-QKV Delta tensors 168 Delta size 0.56 GiB SHA-256 aba8fbaf9ef079a5ca7695c305346c7f1c764dc79435153b6f10352c6594780d The training run froze every parameter outside attention and also froze each attention output projection. The uploaded delta therefore contains exactly the parameters matching "attn" in name and "o_proj" not in name; embeddings, MLP, normalization outside attention, output projections, and LM head are omitted. Frozen tensors from the historical full checkpoint were verified exa