Upload folder using huggingface_hub

.gitattributes

@@ -8,6 +8,7 @@
 *.h5 filter=lfs diff=lfs merge=lfs -text
 *.joblib filter=lfs diff=lfs merge=lfs -text
 *.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.lz4 filter=lfs diff=lfs merge=lfs -text
 *.mlmodel filter=lfs diff=lfs merge=lfs -text
 *.model filter=lfs diff=lfs merge=lfs -text
 *.msgpack filter=lfs diff=lfs merge=lfs -text
@@ -25,11 +26,34 @@
 *.safetensors filter=lfs diff=lfs merge=lfs -text
 saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
 *.tflite filter=lfs diff=lfs merge=lfs -text
 *.tgz filter=lfs diff=lfs merge=lfs -text
 *.wasm filter=lfs diff=lfs merge=lfs -text
 *.xz filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
+*.jsonl filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+# Audio files - uncompressed
+*.pcm filter=lfs diff=lfs merge=lfs -text
+*.sam filter=lfs diff=lfs merge=lfs -text
+*.raw filter=lfs diff=lfs merge=lfs -text
+# Audio files - compressed
+*.aac filter=lfs diff=lfs merge=lfs -text
+*.flac filter=lfs diff=lfs merge=lfs -text
+*.mp3 filter=lfs diff=lfs merge=lfs -text
+*.ogg filter=lfs diff=lfs merge=lfs -text
+*.wav filter=lfs diff=lfs merge=lfs -text
+# Image files - uncompressed
+*.bmp filter=lfs diff=lfs merge=lfs -text
+*.gif filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text
+*.tiff filter=lfs diff=lfs merge=lfs -text
+# JSON files - uncompressed
+*.json filter=lfs diff=lfs merge=lfs -text
+*.log
+# Image files - compressed
+*.jpg filter=lfs diff=lfs merge=lfs -text
+*.jpeg filter=lfs diff=lfs merge=lfs -text
+*.webp filter=lfs diff=lfs merge=lfs -text
+*.gguf filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,27 @@
+Modified MIT License
+
+Copyright (c) 2025 Moonshot AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the “Software”), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+Our only modification part is that, if the Software (or any derivative works
+thereof) is used for any of your commercial products or services that have
+more than 100 million monthly active users, or more than 20 million US dollars
+(or equivalent in other currencies) in monthly revenue, you shall prominently
+display "Kimi K2" on the user interface of such product or service.

README.md

@@ -0,0 +1,254 @@
+# Introduction
+
+**FlagOS** is a unified heterogeneous computing software stack for large models, co-developed with leading global chip manufacturers. With core technologies such as the **FlagScale** distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the FlagOS stack to automatically produce and release various combinations of <chip + open-source model>. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application.
+
+Based on this, the **Kimi-K2-Instruct-FlagOS** model is adapted for the Nvidia chip using the FlagOS software stack, enabling:
+
+### Integrated Deployment
+
+- Deep integration with the open-source [FlagScale framework](https://github.com/FlagOpen/FlagScale)
+- Out-of-the-box inference scripts with pre-configured hardware and software parameters
+- Released **FlagOS**-A800 container image supporting deployment within minutes
+
+### Consistency Validation
+
+- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public.
+
+# Technical Overview
+
+## **FlagScale Distributed Training and Inference Framework**
+
+FlagScale is an end-to-end framework for large models across heterogeneous computing resources, maximizing computational efficiency and ensuring model validity through core technologies. Its key advantages include:
+
+- **Unified Deployment Interface:** Standardized command-line tools support one-click service deployment across multiple hardware platforms, significantly reducing adaptation costs in heterogeneous environments.
+- **Intelligent Parallel Optimization:** Automatically generates optimal distributed parallel strategies based on chip computing characteristics, achieving dynamic load balancing of computation/communication resources.
+- **Seamless Operator Switching:** Deep integration with the FlagGems operator library allows high-performance operators to be invoked via environment variables without modifying model code.
+
+## **FlagGems Universal Large-Model Operator Library**
+
+FlagGems is a Triton-based, cross-architecture operator library collaboratively developed with industry partners. Its core strengths include:
+
+- **Full-stack Coverage**: Over 100 operators, with a broader range of operator types than competing libraries.
+- **Ecosystem Compatibility**: Supports 7 accelerator backends. Ongoing optimizations have significantly improved performance.
+- **High Efficiency**: Employs unique code generation and runtime optimization techniques for faster secondary development and better runtime performance compared to alternatives.
+
+## **FlagEval Evaluation Framework**
+
+FlagEval (Libra)** is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features:
+
+- **Multi-dimensional Evaluation**: Supports 800+ model evaluations across NLP, CV, Audio, and Multimodal fields, covering 20+ downstream tasks including language understanding and image-text generation.
+- **Industry-Grade Use Cases**: Has completed horizontal evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation.
+
+# Evaluation Results
+
+## Benchmark Result
+
+| Metrics   | Kimi-K2-Instruct-FlagOS-H100-CUDA | Kimi-K2-Instruct-FlagOS-FlagOS-Nvidia |
+| --------- | -------------------------------- | ------------------------------------ |
+| AIME      | 0.667                            | 0.700                                |
+| LiveBench | 0.685                            | 0.690                                |
+| MMLU      | 0.773                            | 0.788                                |
+| MUSR      | 0.724                            | 0.710                                |
+
+# User Guide
+
+## General Information 
+
+**Environment Setup**
+
+| System Component                | Version Information                  |
+| ------------------------------- | ------------------------------------ |
+| Docker Version                  | Docker version 24.0.0, build 98fdcd7 |
+| Operating System                | Description:    Ubuntu 20.04 LTS     |
+| FlagScale                       | Version: 0.8.0                       |
+| FlagGems                        | Version: 2.2                         |
+
+## Operation Steps【***need two machines***】
+
+### Download Open-source Model Weights
+
+**Execution under shared storage on master node IP**
+
+```python
+pip install modelscope
+modelscope download --model moonshotai/Kimi-K2-Instruct --local_dir /share/models/Kimi-K2-Instruct
+```
+
+### Download FlagOS Image
+
+**Dual-machine execution**
+
+```python
+docker pull harbor.baai.ac.cn/flagrelease-public/flagrelease_nvidia_kimi_k2
+```
+
+### Start the inference service
+
+**Dual-machine execution**
+
+```
+#Container Startup
+docker run --rm --init --detach \
+  --net=host --uts=host --ipc=host \
+  --security-opt=seccomp=unconfined \
+  --privileged=true \
+  --ulimit stack=67108864 \
+  --ulimit memlock=-1 \
+  --ulimit nofile=1048576:1048576 \
+  --shm-size=32G \
+  -v /share/models:/models \
+  --gpus all \
+  --name flagos \
+  harbor.baai.ac.cn/flagrelease-public/flagrelease_nvidia_kimi_k2 \
+  sleep infinity
+  
+docker exec -it flagos bash
+```
+
+### **Modify configuration files**
+
+**Dual-machine execution**---**Edit the `hostfile.txt` file**
+
+Change the IP in hostfile.txt to the corresponding machine's IP
+
+```python
+vim /root/miniconda3/envs/flagscale-inference/lib/python3.12/site-packages/flag_scale/examples/kimik2/conf/hostfile.txt
+# ip slots type=xxx[optional]
+# master node
+x.x.x.x slots=8 type=gpu 
+# worker nodes
+x.x.x.x slots=8 type=gpu
+```
+
+**Dual-machine execution**---**Modify the `serve.yaml` file**
+
+```python
+vim /root/miniconda3/envs/flagscale-inference/lib/python3.12/site-packages/flag_scale/examples/kimik2/conf/serve.yaml
+```
+
+Modify 
+```
+hostfile: examples/kimik2/conf/hostfile.txt
+```
+to
+```
+hostfile: /root/miniconda3/envs/flagscale-inference/lib/python3.12/site-packages/flag_scale/examples/kimik2/conf/hostfile.txt
+```
+
+Modify 
+```
+USE_FLAGGEMS: false
+```
+to
+```
+USE_FLAGGEMS: true
+```
+
+### **Enter the `flagscale-inference` environment**
+
+**Execution on master node IP**
+
+```python
+conda activate flagscale-inference
+cd /repos/FlagScale
+pip install . -i https://pypi.tuna.tsinghua.edu.cn/simple --no-build-isolation
+```
+
+### **Set up passwordless access from the master container to worker host machines**
+
+```python
+Write the contents of the ~/.ssh/id_rsa.pub file from the flagos container on the master node into the ~/.ssh/authorized_keys file on the worker nodes' physical machines.
+```
+
+### Serve
+
+**Execution on Master Node IP**
+
+```python
+flagscale serve kimik2
+
+#After the service starts, you will see output similar to the following:
+#INFO 07-08 09:49:51 [api_server.py:1349] Starting vLLM API server 0 on http://0.0.0.0:30000
+
+```
+
+# Service Invocation
+
+## API-based Invocation Script
+
+```
+import openai
+openai.api_key = "EMPTY"
+openai.base_url = "http://<server_ip>:30000/v1/"
+model = "Kimi-K2-Instruct-nvidia-origin"
+messages = [
+    {"role": "system", "content": "You are a helpful assistant."},
+    {"role": "user", "content": "What's the weather like today?"}
+]
+response = openai.chat.completions.create(
+    model=model,
+    messages=messages,
+    stream=False,
+)
+for item in response:
+    print(item)
+```
+
+## AnythingLLM Integration Guide
+
+#### 1. Download & Install
+
+- Visit the official site: https://anythingllm.com/
+- Choose the appropriate version for your OS (Windows/macOS/Linux)
+- Follow the installation wizard to complete the setup
+
+#### 2. Configuration
+
+- Launch AnythingLLM
+- Open settings (bottom left, fourth tab)
+- Configure core LLM parameters
+- Click "Save Settings" to apply changes
+
+#### 3. Model Interaction
+
+- After model loading is complete:
+  - Click **"New Conversation"**
+  - Enter your question (e.g., “Explain the basics of quantum computing”)
+  - Click the send button to get a response
+
+# Frequently Asked Questions
+
+### Q1: What should I do if the model fails to load?
+
+- Check if the model weight path is correct.
+- Ensure the  model files are present in the `/models` directory inside the container.
+- Check the container logs: `docker logs flagos`.
+
+### Q2: API call returns a timeout error. What should I do?
+
+- Verify that the server IP address is correct.
+- Check the firewall settings to ensure port 9010 is open.
+- Confirm that the service is running properly: `docker exec flagos ps aux | grep flagscale`.
+
+### Q3: When installing vLLM, if you encounter errors. What should I do?
+
+- You need to retry a few times.
+- Confirm reachability to GitHub.com and associated endpoints.
+- Validate network bandwidth (50MB/s or higher recommended for reliable operation).
+
+# Contributing
+
+We warmly welcome global developers to join us:
+
+1. Submit Issues to report problems
+2. Create Pull Requests to contribute code
+3. Improve technical documentation
+4. Expand hardware adaptation support
+
+# Contact Us
+
+![image](image_.jpeg)
+
+# License
+
+The weights of this model are based on moonshotai/Kimi-K2-Instruct and are open-sourced under the Apache 2.0 License: https://www.apache.org/licenses/LICENSE-2.0.txt.

THIRD_PARTY_NOTICES.md

@@ -0,0 +1,47 @@
+# THIRD_PARTY_NOTICES
+
+This file lists third-party software contained in Kimi-K2 along with their licenses, in compliance with the redistribution clauses of those licenses.
+
+---
+
+## 1. DeepSeek-V3
+
+Our model archietecture is DeepSeek-V3-like. Some of modeling codes are copied from the source repository. 
+
+- **Source Repository**  
+  https://huggingface.co/deepseek-ai/DeepSeek-V3
+
+- **Files / Directories Used**  
+  - configuation_deepseek.py
+  - modeling_deepseek.py
+
+- **License Type**  
+  MIT License
+
+- **Copyright Notice**  
+  Copyright (c)  2023 DeepSeek
+
+- **Full License Text**  
+```
+MIT License
+
+Copyright (c) 2023 DeepSeek
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+```

config.json

@@ -0,0 +1,69 @@
+{
+  "architectures": [
+    "DeepseekV3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_deepseek.DeepseekV3Config",
+    "AutoModel": "modeling_deepseek.DeepseekV3Model",
+    "AutoModelForCausalLM": "modeling_deepseek.DeepseekV3ForCausalLM"
+  },
+  "aux_loss_alpha": 0.001,
+  "bos_token_id": 163584,
+  "eos_token_id": 163585,
+  "first_k_dense_replace": 1,
+  "hidden_act": "silu",
+  "hidden_size": 7168,
+  "initializer_range": 0.02,
+  "intermediate_size": 18432,
+  "kv_lora_rank": 512,
+  "max_position_embeddings": 131072,
+  "model_type": "kimi_k2",
+  "moe_intermediate_size": 2048,
+  "moe_layer_freq": 1,
+  "n_group": 1,
+  "n_routed_experts": 384,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 64,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 61,
+  "num_key_value_heads": 64,
+  "num_nextn_predict_layers": 0,
+  "pretraining_tp": 1,
+  "q_lora_rank": 1536,
+  "qk_nope_head_dim": 128,
+  "qk_rope_head_dim": 64,
+  "quantization_config": {
+    "activation_scheme": "dynamic",
+    "fmt": "e4m3",
+    "quant_method": "fp8",
+    "weight_block_size": [
+      128,
+      128
+    ]
+  },
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 50000.0,
+  "routed_scaling_factor": 2.827,
+  "rope_scaling": {
+    "beta_fast": 1.0,
+    "beta_slow": 1.0,
+    "factor": 32.0,
+    "mscale": 1.0,
+    "mscale_all_dim": 1.0,
+    "original_max_position_embeddings": 4096,
+    "type": "yarn"
+  },
+  "scoring_func": "sigmoid",
+  "seq_aux": true,
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "topk_method": "noaux_tc",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.48.3",
+  "use_cache": true,
+  "v_head_dim": 128,
+  "vocab_size": 163840
+}

configuration.json

@@ -0,0 +1 @@
+{"framework":"Pytorch","task":"text-generation"}

configuration_deepseek.py

@@ -0,0 +1,212 @@
+# Copy from https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/configuration_deepseek.py
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+DEEPSEEK_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
+class DeepseekV3Config(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`DeepseekV3Model`]. It is used to instantiate an DeepSeek
+    model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
+    defaults will yield a similar configuration to that of the DeepSeek-V3.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 129280):
+            Vocabulary size of the Deep model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`DeepseekV3Model`]
+        hidden_size (`int`, *optional*, defaults to 4096):
+            Dimension of the hidden representations.
+        intermediate_size (`int`, *optional*, defaults to 11008):
+            Dimension of the MLP representations.
+        moe_intermediate_size (`int`, *optional*, defaults to 1407):
+            Dimension of the MoE representations.
+        num_hidden_layers (`int`, *optional*, defaults to 32):
+            Number of hidden layers in the Transformer decoder.
+        num_nextn_predict_layers (`int`, *optional*, defaults to 1):
+            Number of nextn predict layers in the DeepSeekV3 Model.
+        num_attention_heads (`int`, *optional*, defaults to 32):
+            Number of attention heads for each attention layer in the Transformer decoder.
+        n_shared_experts (`int`, *optional*, defaults to None):
+            Number of shared experts, None means dense model.
+        n_routed_experts (`int`, *optional*, defaults to None):
+            Number of routed experts, None means dense model.
+        routed_scaling_factor (`float`, *optional*, defaults to 1.0):
+            Scaling factor or routed experts.
+        topk_method (`str`, *optional*, defaults to `gready`):
+            Topk method used in routed gate.
+        n_group (`int`, *optional*, defaults to None):
+            Number of groups for routed experts.
+        topk_group (`int`, *optional*, defaults to None):
+            Number of selected groups for each token(for each token, ensuring the selected experts is only within `topk_group` groups).
+        num_experts_per_tok (`int`, *optional*, defaults to None):
+            Number of selected experts, None means dense model.
+        moe_layer_freq (`int`, *optional*, defaults to 1):
+            The frequency of the MoE layer: one expert layer for every `moe_layer_freq - 1` dense layers.
+        first_k_dense_replace (`int`, *optional*, defaults to 0):
+            Number of dense layers in shallow layers(embed->dense->dense->...->dense->moe->moe...->lm_head).
+                                                            \--k dense layers--/
+        norm_topk_prob (`bool`, *optional*, defaults to False):
+            Whether to normalize the weights of the routed experts.
+        scoring_func (`str`, *optional*, defaults to 'softmax'):
+            Method of computing expert weights.
+        aux_loss_alpha (`float`, *optional*, defaults to 0.001):
+            Auxiliary loss weight coefficient.
+        seq_aux = (`bool`, *optional*, defaults to True):
+            Whether to compute the auxiliary loss for each individual sample.
+        num_key_value_heads (`int`, *optional*):
+            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
+            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
+            `num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
+            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
+            by meanpooling all the original heads within that group. For more details checkout [this
+            paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
+            `num_attention_heads`.
+        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
+            The non-linear activation function (function or string) in the decoder.
+        max_position_embeddings (`int`, *optional*, defaults to 2048):
+            The maximum sequence length that this model might ever be used with.
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        rms_norm_eps (`float`, *optional*, defaults to 1e-06):
+            The epsilon used by the rms normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last key/values attentions (not used by all models). Only
+            relevant if `config.is_decoder=True`.
+        pad_token_id (`int`, *optional*):
+            Padding token id.
+        bos_token_id (`int`, *optional*, defaults to 1):
+            Beginning of stream token id.
+        eos_token_id (`int`, *optional*, defaults to 2):
+            End of stream token id.
+        pretraining_tp (`int`, *optional*, defaults to 1):
+            Experimental feature. Tensor parallelism rank used during pretraining. Please refer to [this
+            document](https://huggingface.co/docs/transformers/parallelism) to understand more about it. This value is
+            necessary to ensure exact reproducibility of the pretraining results. Please refer to [this
+            issue](https://github.com/pytorch/pytorch/issues/76232).
+        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
+            Whether to tie weight embeddings
+        rope_theta (`float`, *optional*, defaults to 10000.0):
+            The base period of the RoPE embeddings.
+        rope_scaling (`Dict`, *optional*):
+            Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
+            strategies: linear and dynamic. Their scaling factor must be a float greater than 1. The expected format is
+            `{"type": strategy name, "factor": scaling factor}`. When using this flag, don't update
+            `max_position_embeddings` to the expected new maximum.
+        attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
+            Whether to use a bias in the query, key, value and output projection layers during self-attention.
+        attention_dropout (`float`, *optional*, defaults to 0.0):
+            The dropout ratio for the attention probabilities.
+
+    ```python
+    >>> from transformers import DeepseekV3Model, DeepseekV3Config
+
+    >>> # Initializing a Deepseek-V3 style configuration
+    >>> configuration = DeepseekV3Config()
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "deepseek_v3"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        vocab_size=129280,
+        hidden_size=7168,
+        intermediate_size=18432,
+        moe_intermediate_size = 2048,
+        num_hidden_layers=61,
+        num_nextn_predict_layers=1,
+        num_attention_heads=128,
+        num_key_value_heads=128,
+        n_shared_experts = 1,
+        n_routed_experts = 256,
+        ep_size = 1,
+        routed_scaling_factor = 2.5,
+        kv_lora_rank = 512,
+        q_lora_rank = 1536,
+        qk_rope_head_dim = 64,
+        v_head_dim = 128,
+        qk_nope_head_dim = 128,
+        topk_method = 'noaux_tc',
+        n_group = 8,
+        topk_group = 4,
+        num_experts_per_tok = 8,
+        moe_layer_freq = 1,
+        first_k_dense_replace = 3,
+        norm_topk_prob = True,
+        scoring_func = 'sigmoid',
+        aux_loss_alpha = 0.001,
+        seq_aux = True,
+        hidden_act="silu",
+        max_position_embeddings=4096,
+        initializer_range=0.02,
+        rms_norm_eps=1e-6,
+        use_cache=True,
+        pad_token_id=None,
+        bos_token_id=0,
+        eos_token_id=1,
+        pretraining_tp=1,
+        tie_word_embeddings=False,
+        rope_theta=10000.0,
+        rope_scaling=None,
+        attention_bias=False,
+        attention_dropout=0.0,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.moe_intermediate_size = moe_intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_nextn_predict_layers = num_nextn_predict_layers
+        self.num_attention_heads = num_attention_heads
+        self.n_shared_experts = n_shared_experts
+        self.n_routed_experts = n_routed_experts
+        self.ep_size = ep_size
+        self.routed_scaling_factor = routed_scaling_factor
+        self.kv_lora_rank = kv_lora_rank
+        self.q_lora_rank = q_lora_rank
+        self.qk_rope_head_dim = qk_rope_head_dim
+        self.v_head_dim = v_head_dim
+        self.qk_nope_head_dim = qk_nope_head_dim
+        self.topk_method = topk_method
+        self.n_group = n_group
+        self.topk_group = topk_group
+        self.num_experts_per_tok = num_experts_per_tok
+        self.moe_layer_freq = moe_layer_freq
+        self.first_k_dense_replace = first_k_dense_replace
+        self.norm_topk_prob = norm_topk_prob
+        self.scoring_func = scoring_func
+        self.aux_loss_alpha = aux_loss_alpha
+        self.seq_aux = seq_aux
+        # for backward compatibility
+        if num_key_value_heads is None:
+            num_key_value_heads = num_attention_heads
+
+        self.num_key_value_heads = num_key_value_heads
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.pretraining_tp = pretraining_tp
+        self.use_cache = use_cache
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.attention_bias = attention_bias
+        self.attention_dropout = attention_dropout
+
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )

generation_config.json

@@ -0,0 +1,4 @@
+{
+  "max_length": 131072,
+  "eos_token_id": 163586
+}

model-1-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23b76842b591c068bc0ad42b3c4aafdc9f4e9a8945b5c0e56a21856078730928
+size 2846451040

model-10-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06eb7cfc383aa668993737ca4a5fa8ebf8ed4272045ef8cdc183baccb4f060ee
+size 17066593104

model-11-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:795003a6b3669ea1f306123e393181938a10f162388582cd9713db0a16394cea
+size 17066595432

model-12-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:10fb0d195b1658d238e0122b328b8e1653bf7c7e80ca4b6525505abd3d648c0d
+size 17066595432

model-13-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e33f1dc9d890f8def7016bcaeb8315b5ba415e7e1dc1a92b7b66ea180dc349cc
+size 17066595432

model-14-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0991a46f6f049eb239edc393d8b064eaf2a0af90a96d0532db66e60ae09c10a4
+size 17066595432

model-15-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29e9499e40e0012c25eedf449a7b37fcb64cfd43e4672b77da30c80445b212db
+size 17066595432

model-16-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5c88baa1759a9f470e9d799b041d383c2ba8ad22b66e81da44d9d302ccedd96
+size 17066595432

model-17-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f484719825079b018a1e6ccd70d056c8dacf036180461dc7f3037dc6de14ca8
+size 17066595432

model-18-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c41a10770609efd56cd6fcc8c0f93d1f3822a35b952cf4ca62cfd6a199f8dd7
+size 17066595432

model-19-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3dec31d95a60d92c61d51a326c50309e94271e56e87448e15000fa7335a497c1
+size 17066595432

model-2-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9698930567abb2e904f78eed4aa6e17c3d7c1e34c3ac73a709cc07787fd37c85
+size 17066593104

model-20-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4cb5b992435aa41c21a311be198c99aac6f97a33ce106ccb90459f6e4a8cd17b
+size 17066595432

model-21-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aad8a72e36b3e1a9f3944e5f25eae492ef3fa6bb864a429802575ec86d1ea2bf
+size 17066595432

model-22-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6dd9104253a6dbf6e8366d8d9f8d6c31465fee8a938775ca6e38b046d9723608
+size 17066595432

model-23-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b63bfb30d3efffe9e33afc7db8f59b4f25f524260900c3fcad426f7196e40b9
+size 17066595432

model-24-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48204d7232ce5abdb7c590ded65ddadd3a36b0fcc5d175cf27827583e1165782
+size 17066595432

model-25-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74efec3d52c9a80b4850f173a7ab9ddb40adc134dabf8e21dd05235c7f797347
+size 17066595432

model-26-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:12d6d8aa724f99aa061a64b6efa533723ee21ebb802900ce1a8b4db18fd16176
+size 17066595432

model-27-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff4e9055eef6c07171fb6b5ee9750def14ca64a56ab391c42123ffd2cf4b7828
+size 17066595432

model-28-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6757464954cbbf6bc8046dd62628e008de48ba3ae50f849a2187bcf873f84336
+size 17066595432

model-29-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c6db8e79ecbd4b6e198f8543ac4ad4d0e0954706926c992b7f1ff7432453698
+size 17066595432

model-3-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1391b1ba27e5cf04ce0b18f45bb0a3c82fbf5433ce0af87d296c1d5a1fd56a7
+size 17066593104

model-30-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:918dd637c2a7590bfe113a32f93ab6a3cd9c68abdec22a6b54ca3c0b0a0dc88b
+size 17066595432

model-31-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44bf952854075f1eaf159d7ae35bd58bc2aa7beab42e16bcef3613c1493273c8
+size 17066595432

model-32-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c5f37676bd435e24a234b4164f59b4559606d6d192f65f97c89b4c6e6838b11a
+size 17066595432

model-33-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2741414690956a950e3abe49eba9c9ba61fabce57ef0ec123dd0007a9cfdd73c
+size 17066595432

model-34-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce15a4c296ca738960dcadcde4796ad7c95fcdc503f5557dab1bccc8f0e4a427
+size 17066595432

model-35-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e663c0fdbae60daa246e2dec50c5e892289d53f4ccae63ad977e61c78aa5b99
+size 17066595432

model-36-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ab798ec5e9d98552293eb73cc429faac320caa82856b7303f4487217214d00a
+size 17066595432

model-37-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:97899b584b7cfc65e1adc601cb79f1b60533389ab02cf4409ed72e4ecc7a485c
+size 17066595432

model-38-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5707040ecd0aad270628ee21f57b87adecd76fbe82d95a9ddfe536d0b4628e47
+size 17066595432

model-39-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6207e2ba874f7aecc77850cf21d747201514b626ed76f3b65064f3fd4b9cf574
+size 17066595432

model-4-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5226ae5591c4fba1bb42960e197b645d7bbb5fb5e8386c62d453a12a3fd4b715
+size 17066593104

model-40-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc086b326aad450fef9dc1bee9d06e894f9a93bd1af2d7c1ad58ec6661c12b6c
+size 17066595432

model-41-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6defa0ef782e8db6d48417126b19b971e7fd970d6dca41cca42e110be31709a1
+size 17066595432

model-42-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3964b435c12e2b076f815b6c798c447a5f1b5368b450cd92663486b152edf26
+size 17066595432

model-43-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83658a4ed7f1329cf7b41176a89deac5fbac1f8da02b2d61cd0f5dcb2bb531fc
+size 17066595432

model-44-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac318d216a6e534acd96fc6bbefb7d666d005a7d389ba7a5ad7494a87d7a95fd
+size 17066595432

model-45-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82c1ddd20f07e712205f1a3cf6d9f44ad84f405f63a8a77f9e11fe936538c028
+size 17066595432

model-46-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:10e59e5c4e25e9ebc90c927d1d494a9c212cdfb6525ac89d3e56b02409816673
+size 17066595432

model-47-of-61.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20d38a14cae31b03e32e4a4edec78c3447576f171d9e1a7e9b7aef92581580c6
+size 17066595432