Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+model.onnx.data filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,228 @@
+---
+license: apache-2.0
+tags:
+- small-language-model
+- jee
+- exam-centric
+- indian-education
+- reinforcement-learning
+- supervised-finetuning
+- model-merging
+- rejection-sampling
+- mathematics
+- ai4education
+- physicswallah
+- onnx
+- onnxruntime-genai
+- onnxruntime
+language:
+- en
+library_name: onnxruntime-genai
+base_model_relation: quantized
+base_model: Prince-1/Aryabhata-1.0
+pipeline_tag: text-generation
+model_creator: Physics Wallah AI Research
+model_type: Causal decoder-based model
+---
+
+# Aryabhatta 1.0 : An exam-focused language model for JEE Math
+
+![](benchmark.png)
+
+## Overview
+
+**Aryabhata 1.0** is a 7B parameter small language model for mathematics developed by **Physics Wallah AI Research**, optimized for high-stakes Indian competitive exams like **JEE Mains**. Despite its compact size, Aryabhata 1.0 achieves **state-of-the-art performance** on exam-centric reasoning tasks with impressive **token efficiency** and low inference cost.
+
+
+> 🚧 *Aryabhata 1.0 is an **experimental release**. We are actively seeking feedback — please contribute in the Discussion tab of this repo.*
+---
+
+## 🧠 Key Features
+
+- **Architecture**: 7B parameter causal decoder-based model.
+- **Exam-Centric Optimization**: Specifically tuned for JEE-level Mathematics reasoning.
+- **High Accuracy**:
+  - **86%** on **JEE Mains January 2025** session.
+  - **90.2%** on **JEE Mains April 2025** session.
+- **Token Efficiency**: Operates effectively around a **~2K token window**, compared to ~8K required by other reasoning models.
+- **Compute Efficient**: Trained on a **1x2 NVIDIA H100 GPU** using optimized pipeline.
+
+---
+
+## 🛠️ Training Details
+
+- **Training Data**: ~130K problem-solution pairs curated from proprietary Physics Wallah exam datasets.
+- **Training Pipeline**:
+  - **Model Merging**
+  - **Rejection Sampling**
+  - **Supervised Fine-Tuning (SFT)**
+  - **Reinforcement Learning with Verifiable Rewards (RLVR)**
+
+### 🔀 Model Merging
+We began with model merging (Weighted average) to build a strong initialization (Aryabhata 0.5) by combining diverse model capabilities:
+* Qwen 2.5 Math: A robust math-centric LLM with solid symbolic math foundations.
+* Ace Math: An enhanced version of Qwen 2.5 Math, fine-tuned by NVIDIA for improved accuracy in mathematics benchmarks.
+* DeepSeek R1 Distill Qwen: A long-form reasoning model, fine-tuned on reasoning traces distilled from DeepSeek R1.
+
+### 📚 Data Curation + Rejection Sampling
+We extracted ~250K raw questions from Physics Wallah's internal database and applied aggressive filtering and cleaning:
+* Removed: diagram-based, non-English, and option-heavy questions.
+* Kept: questions matching the distribution of JEE Main 2019–2024.
+Final curated dataset: ~130K high-quality questions.
+
+For each question:
+* Generated 4 CoTs using Aryabhata 0.5.
+* Retained only those leading to correct final answers.
+
+Resulting Dataset:
+* ~100K questions
+* ~350K high-quality CoTs
+
+We used this dataset for SFT.
+
+### 🎯 Reinforcement Learning with Verifiable Rewards (RLVR)
+We used a custom in-house variant of Group Relative Policy Optimization (GRPO), adapted for math-specific reward functions.
+* Removed KL-divergence penalty
+* Removed clipping
+
+We used RLVR on the remaining ~30K questions.
+
+This multi-phase training strategy allows Aryabhata 1.0 to capture **pedagogy-aligned reasoning patterns**, making it highly effective for solving real student queries in mathematics.
+
+---
+
+## 📊 Performance Highlights
+
+### Evaluation Setup
+All evaluations were performed with temperature = 0.0, and we report pass@1 accuracy.
+
+#### Evaluation Datasets
+We evaluated the model on two sets of official JEE Mains 2025 mathematics papers:
+* January Session: 10 question papers containing 250 questions.
+* April Session: 9 question papers containing 225 questions.
+
+Each paper includes a mix of:
+* Multiple Choice Questions (MCQs) with one correct option
+* Numeric Answer Type (NAT) questions requiring precise numerical responses
+
+#### Evaluation Metric
+We used a composite evaluation metric to reflect real-world grading rigor and reduce false positives:
+
+1. Float Match
+  * Compares predicted and target answers within a tolerance (±1e-9)
+  * Handles rounding artifacts and small numerical errors robustly
+2. String Match
+  * Used for symbolic answers (e.g., fractions, radicals)
+  * Uses strict exact match — predictions must match ground truth character-for-character
+3. LLM-as-Judge (GPT-4o-mini)
+  * Used for Mathematical equivalence for ambiguous formats
+
+### 🔹 Accuracy Comparison Across Models
+![](accuracy.png)
+> *Aryabhata has the best accuracy on JEE Main Maths, on par with frontier models*
+
+### 🔹 Accuracy vs Token Usage
+![](accuracy-vs-token.png)
+> *Aryabhata is on par with frontier models in terms of accuracy vs token usage*
+
+---
+
+## 🔧 Intended Use
+
+**Primary Use Cases**:
+- Competitive exam preparation (JEE Main level mathematics problems)
+- Question answering and doubt-solving systems
+- Educational tutoring and concept explanation
+
+
+## 💡 How to Use
+
+### 🧪 Using with 🤗 Transformers
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
+
+model_id = "PhysicsWallahAI/Aryabhata-1.0"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id)
+
+
+# Define stop strings
+stop_strings = ["<|im_end|>", "<|end|>", "<im_start|>", "⁠```python\n", "⁠<|im_start|>", "]}}]}}]"]
+
+def strip_bad_tokens(s, stop_strings):
+    for suffix in stop_strings:
+        if s.endswith(suffix):
+            return s[:-len(suffix)]
+    return s
+
+
+# Create generation config (can also set temperature, top_p, etc.)
+generation_config = GenerationConfig(
+    max_new_tokens=4096,
+    stop_strings = stop_strings
+)
+
+query = 'Find all the values of \\sqrt[3]{1}'
+messages = [{'role': 'system', 'content': 'Think step-by-step; put only the final answer inside \\boxed{}.'},
+            {'role': 'user', 'content': query}]
+
+text = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True
+)
+inputs = tokenizer([text], return_tensors="pt")
+outputs = model.generate(**inputs, generation_config=generation_config, tokenizer=tokenizer)
+
+print(strip_bad_tokens(tokenizer.decode(outputs[0], skip_special_tokens=True), stop_strings))
+````
+
+---
+
+### ⚡ Using with vLLM
+
+To run the model efficiently using vLLM:
+
+```python
+from vllm import LLM, SamplingParams
+
+# Initialize model (downloads from Hugging Face if not local)
+llm = LLM(model="PhysicsWallahAI/Aryabhata-1.0")
+
+# Define prompt and sampling configuration
+query = 'Find all the values of \\sqrt[3]{1}'
+messages = [{'role': 'system', 'content': 'Think step-by-step; put only the final answer inside \\boxed{}.'},
+            {'role': 'user', 'content': query}]
+sampling_params = SamplingParams(temperature=0.0, max_tokens=4*1024, stop=["<|im_end|>", "<|end|>", "<im_start|>", "⁠```python\n", "⁠<|im_start|>", "]}}]}}]"])
+
+# Run inference
+results = llm.chat(messages, sampling_params)
+
+# Print result
+print(results[0].outputs[0].text.strip())
+```
+
+---
+
+## 🚀 Roadmap
+
+**Aryabhata 2.0** (Upcoming):
+- Extending domain coverage to **Physics** and **Chemistry**
+- Supporting **JEE Advanced**, **NEET**, and **Foundation syllabus**
+- Further optimization for affordability and accuracy in real-time deployments
+
+---
+
+## 🤝 Citation
+
+If you use this model, please cite:
+
+```bibtex
+@misc{Aryabhata2025,
+  title = {Aryabhata 1.0: A compact, exam-focused language model tailored for mathematics in Indian competitive exams, especially JEE Main.},
+  author = {Physics Wallah AI Research},
+  year = {2025},
+  note = {\url{https://huggingface.co/PhysicsWallahAI/Aryabhata-1.0}},
+}

chat_template.jinja

@@ -0,0 +1,54 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'Please reason step by step, and put your final answer within \\boxed{}.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nPlease reason step by step, and put your final answer within \\boxed{}.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

genai_config.json

@@ -0,0 +1,50 @@
+{
+    "model": {
+        "bos_token_id": 151643,
+        "context_length": 131072,
+        "decoder": {
+            "session_options": {
+                "log_id": "onnxruntime-genai",
+                "provider_options": []
+            },
+            "filename": "model.onnx",
+            "head_size": 128,
+            "hidden_size": 3584,
+            "inputs": {
+                "input_ids": "input_ids",
+                "attention_mask": "attention_mask",
+                "position_ids": "position_ids",
+                "past_key_names": "past_key_values.%d.key",
+                "past_value_names": "past_key_values.%d.value"
+            },
+            "outputs": {
+                "logits": "logits",
+                "present_key_names": "present.%d.key",
+                "present_value_names": "present.%d.value"
+            },
+            "num_attention_heads": 28,
+            "num_hidden_layers": 28,
+            "num_key_value_heads": 4
+        },
+        "eos_token_id": 151643,
+        "pad_token_id": 151643,
+        "type": "qwen2",
+        "vocab_size": 152064
+    },
+    "search": {
+        "diversity_penalty": 0.0,
+        "do_sample": false,
+        "early_stopping": true,
+        "length_penalty": 1.0,
+        "max_length": 131072,
+        "min_length": 0,
+        "no_repeat_ngram_size": 0,
+        "num_beams": 1,
+        "num_return_sequences": 1,
+        "past_present_share_buffer": false,
+        "repetition_penalty": 1.0,
+        "temperature": 1.0,
+        "top_k": 1,
+        "top_p": 1.0
+    }
+}

model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:395e73c4753dfcd56a866f02704e187020507a2c3312cb3bcbe56465000f5100
+size 688147

model.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e4dca4d35cceb00909023e30a767da00b18af004a05844639813fc283f093e8
+size 15264787456

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<｜begin▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e20ddafc659ba90242154b55275402edeca0715e5dbb30f56815a4ce081f4893
+size 11422778

tokenizer_config.json

@@ -0,0 +1,194 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<｜end▁of▁sentence｜>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<｜User｜>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151645": {
+      "content": "<｜Assistant｜>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151646": {
+      "content": "<｜begin▁of▁sentence｜>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|EOT|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151648": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151649": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<｜end▁of▁sentence｜>",
+  "extra_special_tokens": {},
+  "legacy": true,
+  "model_max_length": 16384,
+  "pad_token": "<｜end▁of▁sentence｜>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizerFast",
+  "unk_token": null,
+  "use_default_system_prompt": false
+}