upstage/solar-pro4-tokenizer开源了新模型

Upstage upstage/solar-pro4-tokenizer upstage/solar-pro4-tokenizer https://huggingface.co/upstage/solar-pro4-tokenizer

upstage/solar-pro4-tokenizer · Hugging Face Hugging Face (https://huggingface.co/) Models (https://huggingface.co/models) Datasets (https://huggingface.co/datasets) Spaces (https://huggingface.co/spaces) Buckets new (https://huggingface.co/storage) Docs (https://huggingface.co/docs) Enterprise (https://huggingface.co/enterprise) Pricing (https://huggingface.co/pricing) Website Tasks (https://huggingface.co/tasks) HuggingChat (https://huggingface.co/chat) Collections (https://huggingface.co/collections) Languages (https://huggingface.co/languages) Organizations (https://huggingface.co/organizations) Community Blog (https://huggingface.co/blog) Posts (https://huggingface.co/posts) Daily Papers (https://huggingface.co/papers) Hardware (https://huggingface.co/hardware) Learn (https://huggingface.co/learn) Discord (https://huggingface.co/join/discord) Forum (https://discuss.huggingface.co/) GitHub (https://github.com/huggingface) Solutions Team & Enterprise (https://huggingface.co/enterprise) Hugging Face PRO (https://huggingface.co/pro) Enterprise Support (https://huggingface.co/support) Inference Providers (https://huggingface.co/inference/models) Inference Endpoints (https://huggingface.co/inference-endpoints) Storage Buckets (https://huggingface.co/storage) Log In (https://huggingface.co/login) Sign Up (https://huggingface.co/join) (https://huggingface.co/upstage) upstage (https://huggingface.co/upstage) / solar-pro4-tokenizer (https://huggingface.co/upstage/solar-pro4-tokenizer) Like 1 Follow upstage 1.21k English (https://huggingface.co/models?language=en) Korean (https://huggingface.co/models?language=ko) Japanese (https://huggingface.co/models?language=ja) tokenizer (https://huggingface.co/models?other=tokenizer) License: upstage-solar-license Model card (https://huggingface.co/upstage/solar-pro4-tokenizer) Files Files and versions xet (https://huggingface.co/upstage/solar-pro4-tokenizer/tree/main) Community (https://huggingface.co/upstage/solar-pro4-tokenizer/discussions) Copy to bucket new Solar Pro 4 Tokenizer (https://huggingface.co/upstage/solar-pro4-tokenizer#solar-pro-4-tokenizer) Usage (https://huggingface.co/upstage/solar-pro4-tokenizer#usage) License (https://huggingface.co/upstage/solar-pro4-tokenizer#license) (https://huggingface.co/upstage/solar-pro4-tokenizer#solar-pro-4-tokenizer) Solar Pro 4 Tokenizer Tokenizer for Upstage Solar Pro 4, with a vocabulary of 196,608 tokens and support for English, Korean, Japanese, and more. This repository contains tokenizer files for encoding text and estimating text token counts. It does not contain model weights or a chat template. (https://huggingface.co/upstage/solar-pro4-tokenizer#usage) Usage Install the Transformers library: pip install transformers from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("upstage/solar-pro4-tokenizer") text = "Hi, how are you?" token_ids = tokenizer.encode(text, add_special_tokens=False) print("Token IDs:", token_ids) print("Tokens:", tokenizer.convert_ids_to_tokens(token_ids)) print("Number of tokens:", len(token_ids)) print("Decoded text:", tokenizer.decode(token_ids, skip_special_tokens=False)) To load a downloaded copy without network access: from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained( "./solar-pro4-tokenizer", local_files_only=True, trust_remote_code=False, ) These examples count tokens in the supplied text. A full API request can also include message formatting, system instructions, and tool definitions. Use the API response's usage fields for the actual request token usage. See the token counting guide (https://console.upstage.ai/docs/guides/counting-tokens) for more information. (https://huggingface.co/upstage/solar-pro4-tokenizer#license) License See Upstage Solar License (https://huggingface.co/upstage/solar-pro4-tokenizer/tree/main/LICENSE) . Downloads last month - Downloads are not tracked for this m