Update README.md

README.md

@@ -1,3 +1,209 @@
----
-license: mit
----
+---
+license: mit
+datasets:
+- yahma/alpaca-cleaned
+---
+## Model Details
+
+This model builds upon the neuromorphic **Llama-SNN-LTC** base architecture, incorporating **Spiking Neural Networks (SNNs)** and **Liquid Time Constants (LTCs)**, and fine-tunes it specifically for instruction following using the Alpaca Cleaned dataset.
+
+**Model Type**: Instruction-Following Language Model with Neuromorphic Enhancements  
+**Supported Languages**: English  
+**Number of Parameters**: 155.8M  
+**Context Length**: 1024 tokens  
+**Base Architecture**: Llama with SNN/LTC modifications  
+**Base Model**: rootxhacker/arthemis-lm  
+**Fine-tuning Data**: Alpaca Cleaned (~52K instruction-response pairs)
+
+### Architecture Features
+- **Spiking Neural Networks** in attention mechanisms for temporal processing
+- **Liquid Time Constants** in feed-forward layers for adaptive dynamics
+- **12-layer transformer backbone** with neuromorphic enhancements
+- **RoPE positional encoding** for sequence understanding
+- **Custom surrogate gradient training** for differentiable spike computation
+- **Instruction-following fine-tuning** for enhanced conversational abilities
+
+Here are my major model configurations:
+
+```
+hidden_size = 768
+intermediate_size = 2048
+num_hidden_layers = 12
+num_attention_heads = 12
+num_key_value_heads = 12
+max_position_embeddings = 1024
+vocab_size = 50257
+spiking_threshold = 1.0
+ltc_hidden_size = 256
+ltc_layers = 2
+```
+
+## Usage
+
+### Install dependencies
+```bash
+pip install transformers torch numpy
+```
+
+### Run code!
+```python
+# Note: This model requires custom implementation due to SNN/LTC architecture
+# Standard transformers library cannot load this model directly
+
+# For custom loading, you'll need the specialized architecture:
+from custom_model import LlamaSNNLTCModel
+from transformers import AutoTokenizer
+
+# Load tokenizer
+tokenizer = AutoTokenizer.from_pretrained("microsoft/DialoGPT-small")
+tokenizer.pad_token = tokenizer.eos_token
+
+# Load the instruction-tuned model
+model = LlamaSNNLTCModel.from_pretrained("rootxhacker/arthemis-instruct")
+
+# For instruction-following generation
+def generate_instruction_response(instruction, input_text="", model=None, tokenizer=None, max_length=150):
+    model.eval()
+    device = next(model.parameters()).device
+    
+    # Reset model states for clean generation
+    model.reset_states()
+    
+    # Format prompt in Alpaca style
+    if input_text.strip():
+        prompt = f"### Instruction:\n{instruction}\n\n### Input:\n{input_text}\n\n### Response:\n"
+    else:
+        prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
+    
+    inputs = tokenizer(prompt, return_tensors='pt').to(device)
+    input_ids = inputs['input_ids']
+    
+    with torch.no_grad():
+        for _ in range(max_length - input_ids.shape[1]):
+            outputs = model(input_ids)
+            logits = outputs['logits'][0, -1, :]
+            
+            # Sample with temperature for more natural responses
+            logits = logits / 0.7
+            probs = torch.softmax(logits, dim=-1)
+            next_token = torch.multinomial(probs, 1)
+            
+            input_ids = torch.cat([input_ids, next_token.unsqueeze(0)], dim=-1)
+            
+            if next_token.item() == tokenizer.eos_token_id:
+                break
+    
+    generated = tokenizer.decode(input_ids[0], skip_special_tokens=True)
+    
+    # Extract just the response part
+    if "### Response:\n" in generated:
+        response = generated.split("### Response:\n")[-1].strip()
+        return response
+    
+    return generated
+
+# Example usage
+instruction = "Explain what artificial intelligence is in simple terms."
+response = generate_instruction_response(instruction, model=model, tokenizer=tokenizer)
+print(f"Instruction: {instruction}")
+print(f"Response: {response}")
+```
+
+
+## Evaluation
+
+I performed evaluation using the standard lm-evaluation-harness setup. Following similar methodology to TinyLlama and MicroLlama, I used acc_norm for most datasets except for winogrande and boolq which used acc as the metrics.
+
+### Results Comparison
+
+| Model | Params | Budget | HellaSwag | OBQA | WinoGrande | ARC_e | ARC_c | BoolQ | Avg |
+|-------|--------|--------|-----------|------|------------|-------|-------|-------|-----|
+| **rootxhacker/arthemis-lm** | **155.8M** | **<$50** | **24.65** | **20.60** | **48.10** | **28.20** | **22.20** | **39.80** | **30.59** |
+| google/bert-large-uncased | 336M | N/A | 24.53 | 26.20 | 49.80 | 25.08 | 25.68 | 40.86 | 32.03 |
+
+
+## Technical Specifications
+
+```
+Architecture: Llama + Spiking Neural Networks + Liquid Time Constants
+Hidden Size: 768
+Intermediate Size: 2048
+Attention Heads: 12
+Layers: 12
+Max Position Embeddings: 1024
+Vocabulary Size: 50,257
+Spiking Threshold: 1.0
+LTC Hidden Size: 256
+Training Precision: FP32
+Fine-tuning Dataset: Alpaca Cleaned (52K instructions)
+```
+
+## Training Details
+
+The model was fine-tuned from rootxhacker/arthemis-lm using:
+- **Base Model**: rootxhacker/arthemis-lm (pretrained neuromorphic LLM)
+- **Dataset**: Alpaca Cleaned (~52K instruction-response pairs)
+- **Hardware**: Google Colab Pro Plus (A100 GPU)
+- **Training Steps**: 5,000 steps
+- **Batch Size**: 4 with gradient accumulation
+- **Learning Rate**: 5e-5 (lower for fine-tuning)
+- **Precision**: FP32 for stability with neuromorphic components
+
+### Key Features
+- **Instruction Format**: Uses Alpaca's structured instruction format
+- **Response Generation**: Optimized for helpful, accurate responses
+- **Neuromorphic Preservation**: Maintains SNN/LTC benefits during fine-tuning
+- **Budget-Conscious**: Additional fine-tuning cost under $10
+
+## Fine-tuning Process
+
+The fine-tuning process involved:
+1. **Base Model Loading**: Started from the pretrained arthemis-lm checkpoint
+2. **Data Formatting**: Converted Alpaca instructions to proper format
+3. **Careful Training**: Lower learning rate to preserve base model knowledge
+4. **State Management**: Proper handling of SNN/LTC states during training
+5. **Validation**: Continuous monitoring of instruction-following quality
+
+
+## Limitations
+
+- **Training Data**: Limited to Alpaca Cleaned dataset scope
+- **Context Length**: Maximum 1024 tokens
+- **Domain**: Primarily English instructions
+- **Custom Architecture**: Requires specialized loading code
+- **Scale**: Smaller than commercial instruction models
+
+## Model Sources
+
+- **Repository**: [Coming Soon]
+- **Base Model**: [rootxhacker/arthemis-lm](https://huggingface.co/rootxhacker/arthemis-lm)
+- **Hugging Face**: [rootxhacker/arthemis-instruct](https://huggingface.co/rootxhacker/arthemis-instruct)
+
+## Future Work
+
+- Scale instruction dataset for broader capabilities
+- Add multi-turn conversation support
+- Implement reinforcement learning from human feedback (RLHF)
+- Explore specialized instruction types (coding, math, reasoning)
+- Compare instruction-following efficiency with standard transformers
+
+## Acknowledgments
+
+Special thanks to **keeeeenw** for the inspiration and open-source MicroLlama project, which demonstrated that impressive language models can be built on a budget. This work extends those principles to instruction-following capabilities while exploring neuromorphic computing approaches.
+
+Thanks to the Stanford Alpaca team for the high-quality instruction dataset that made this fine-tuning possible.
+
+## Citation
+
+```bibtex
+@misc{arthemis-instruct-2024,
+  title={Arthemis-Instruct: A Neuromorphic Instruction-Following Model with Spiking Neural Networks and Liquid Time Constants},
+  author={rootxhacker},
+  year={2024},
+  howpublished={\url{https://huggingface.co/rootxhacker/arthemis-instruct}}
+}
+```
+
+## License
+
+Apache License 2.0