README.md

---
library_name: transformers
tags: 
  - nucleotide-transformer
  - AMR-prediction
  - bioinformatics
  - sequence-classification
  - LoRA
---

# Model Card for DraGNOME-2.5b-v1

This model is a fine-tuned version of the Nucleotide Transformer (2.5B parameters, multi-species) for Antimicrobial Resistance (AMR) prediction, optimized for handling class imbalance and training efficiency.

## Model Details

### Model Description

This model is a fine-tuned version of InstaDeepAI's Nucleotide Transformer (2.5B parameters, multi-species) designed for binary classification of nucleotide sequences to predict Antimicrobial Resistance (AMR). It leverages LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning and includes optimizations for class imbalance and training efficiency, with checkpointing to handle Google Colab's 24-hour runtime limit. The model was trained on a dataset of positive (AMR) and negative (non-AMR) sequences.

- **Developed by:** Blaise Alako
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** alakob
- **Model type:** Sequence Classification
- **Language(s) (NLP):** Nucleotide sequences
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** InstaDeepAI/nucleotide-transformer-2.5b-multi-species

### Model Sources [optional]

- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]

## Uses

### Direct Use

This model can be used directly for predicting whether a given nucleotide sequence is associated with Antimicrobial Resistance (AMR) without additional fine-tuning.

### Downstream Use 

The model can be further fine-tuned for specific AMR-related tasks or integrated into larger bioinformatics pipelines for genomic analysis.

### Out-of-Scope Use

The model is not intended for general-purpose sequence analysis beyond AMR prediction, nor for non-biological sequence data. Misuse could include applying it to unrelated classification tasks where its training data and architecture are not applicable.

## Bias, Risks, and Limitations

The model may exhibit bias due to imbalances in the training dataset or underrepresentation of certain AMR mechanisms. It is limited by the quality and diversity of the training sequences and may not generalize well to rare or novel AMR variants.

### Recommendations

Users (both direct and downstream) should be made aware of the risks, biases, and limitations of the model. Validation on diverse datasets and careful interpretation of predictions are recommended.

## How to Get Started with the Model

Use the code below to get started with the model:

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import get_peft_model, LoraConfig

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("InstaDeepAI/nucleotide-transformer-2.5b-multi-species")
model = AutoModelForSequenceClassification.from_pretrained("alakob/DraGNOME-2.5b-v1")

# Example inference
sequence = "ATGC..."  # Replace with your nucleotide sequence
inputs = tokenizer(sequence, truncation=True, max_length=1000, return_tensors="pt")
outputs = model(**inputs)
prediction = outputs.logits.argmax(-1).item()  # 0 = non-AMR, 1 = AMR
```

## Training Details

### Training Data

- **Negative sequences (non-AMR):**  
  `DSM_20231.fasta`, `ecoli-k12.fasta`, `FDA.fasta`

- **Positive sequences (AMR):**  
  `28227009.fasta`, `nucleotide_fasta_protein_homolog_model_variants.fasta`, `40859916.fasta`,  
  `nucleotide_fasta_protein_overexpression_model_variants.fasta`, `all_resfinder.fasta`,  
  `nucleotide_fasta_protein_variant_model_variants.fasta`, `efaecium.fasta`,  
  `nucleotide_fasta_rRNA_gene_variant_model_variants.fasta`

### Training Procedure

#### Preprocessing

Sequences were tokenized using the Nucleotide Transformer tokenizer with a maximum length of 1000 tokens and truncation applied where necessary.

#### Training Hyperparameters

- **Training regime:** fp16 mixed precision  
- **Learning rate:** 5e-5  
- **Batch size:** 8 (with gradient accumulation steps = 8)  
- **Epochs:** 10  
- **Optimizer:** AdamW (default in Hugging Face Trainer)  
- **Scheduler:** Linear with 10% warmup  
- **LoRA parameters:** `r=32`, `alpha=64`, `dropout=0.1`, `target_modules=["query", "value"]`

#### Speeds, Sizes, Times

Training was performed on Google Colab with checkpointing every 500 steps, retaining the last 3 checkpoints.  
Exact throughput and times depend on Colab's hardware allocation NVIDIA A100 GPU.

---

## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data

The test set was derived from a 10% split of the DraGNOME-2.5b-v1 dataset, stratified by AMR labels.

#### Factors

Evaluation was performed across AMR and non-AMR classes.

#### Metrics

- **Accuracy:** Proportion of correct predictions  
- **F1 Score:** Harmonic mean of precision and recall (primary metric)  
- **Precision:** Positive predictive value  
- **Recall:** Sensitivity  
- **ROC-AUC:** Area under the receiver operating characteristic curve

### Results

[More Information Needed]

#### Summary

[More Information Needed]

---

## Model Examination [optional]

[More Information Needed]

---

## Environmental Impact

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** Google Colab NVIDIA A100 GPU  
- **Hours used:** [More Information Needed]  
- **Cloud Provider:** Google Colab  
- **Compute Region:** [More Information Needed]  
- **Carbon Emitted:** [More Information Needed]

---

## Technical Specifications [optional]

### Model Architecture and Objective

The model uses the Nucleotide Transformer architecture (2.5B parameters) with a sequence classification head, fine-tuned with LoRA for AMR prediction.

### Compute Infrastructure

Training was performed on Google Colab with persistent storage via Google Drive.

#### Hardware

- NVIDIA A100 GPU 

#### Software

- Transformers (Hugging Face)  
- PyTorch  
- PEFT (Parameter-Efficient Fine-Tuning)  
- Weights & Biases (wandb) for logging

---

## Citation [optional]

**BibTeX:**  
[More Information Needed]

**APA:**  
[More Information Needed]

---

## Glossary 

- **AMR:** Antimicrobial Resistance  
- **LoRA:** Low-Rank Adaptation  
- **Nucleotide Transformer:** A transformer-based model for nucleotide sequence analysis

---

## More Information [optional]

[More Information Needed]

---

## Model Card Authors

Blaise Alako

---

## Model Card Contact

[More Information Needed]