Update README.md

README.md

@@ -1,8 +1,216 @@
-# Azerbaijani NER Model
-        
-This is a Named Entity Recognition model for Azerbaijani language based on XLM-RoBERTa.
-
-## Model details
-- Base model: xlm-roberta-base
-- Trained for Named Entity Recognition
-- Language: Azerbaijani
+---
+license: cc-by-4.0
+language:
+- az
+base_model:
+- FacebookAI/xlm-roberta-base
+pipeline_tag: token-classification
+tags:
+- personally identifiable information
+- pii
+- ner
+- azerbaijan
+---
+
+
+# PII NER Azerbaijani v2
+
+**PII NER Azerbaijani** is a second version of fine-tuned Named Entity Recognition (NER) model (First version: <a target="_blank" href="https://huggingface.co/LocalDoc/private_ner_azerbaijani">PII NER Azerbaijani</a>) based on XLM-RoBERTa. 
+It is trained on Azerbaijani pii data for classification personally identifiable information such as names, dates of birth, cities, addresses, and phone numbers from text.
+
+## Model Details
+
+- **Base Model:** XLM-RoBERTa
+- **Training Metrics:**
+- 
+| Epoch | Training Loss | Validation Loss | Precision | Recall  | F1       |
+|-------|----------------|------------------|-----------|---------|----------|
+| 1     | 0.029100       | 0.025319         | 0.963367  | 0.962449| 0.962907 |
+| 2     | 0.019900       | 0.023291         | 0.964567  | 0.968474| 0.966517 |
+| 3     | 0.015400       | 0.018993         | 0.969536  | 0.967555| 0.968544 |
+| 4     | 0.012700       | 0.017730         | 0.971919  | 0.969768| 0.970842 |
+| 5     | 0.011100       | 0.018095         | 0.973056  | 0.970075| 0.971563 |
+
+
+
+- **Test Metrics:**  
+
+- **Precision:** 0.9760  
+- **Recall:** 0.9732  
+- **F1 Score:** 0.9746  
+
+
+## Detailed Test Classification Report
+
+| Entity              | Precision | Recall | F1-score | Support |
+|---------------------|-----------|--------|----------|---------|
+| AGE                 | 0.98      | 0.98   | 0.98     | 509     |
+| BUILDINGNUM         | 0.97      | 0.75   | 0.85     | 1285    |
+| CITY                | 1.00      | 1.00   | 1.00     | 2100    |
+| CREDITCARDNUMBER    | 0.99      | 0.98   | 0.99     | 249     |
+| DATE                | 0.85      | 0.92   | 0.88     | 1576    |
+| DRIVERLICENSENUM    | 0.98      | 0.98   | 0.98     | 258     |
+| EMAIL               | 0.98      | 1.00   | 0.99     | 1485    |
+| GIVENNAME           | 0.99      | 1.00   | 0.99     | 9926    |
+| IDCARDNUM           | 0.99      | 0.99   | 0.99     | 1174    |
+| PASSPORTNUM         | 0.99      | 0.99   | 0.99     | 426     |
+| STREET              | 0.94      | 0.98   | 0.96     | 1480    |
+| SURNAME             | 1.00      | 1.00   | 1.00     | 3357    |
+| TAXNUM              | 0.99      | 1.00   | 0.99     | 240     |
+| TELEPHONENUM        | 0.97      | 0.95   | 0.96     | 2175    |
+| TIME                | 0.96      | 0.96   | 0.96     | 2216    |
+| ZIPCODE             | 0.97      | 0.97   | 0.97     | 520     |
+
+
+### Averages
+
+| Metric        | Precision | Recall | F1-score | Support |
+|---------------|-----------|--------|----------|---------|
+| **Micro avg** | 0.98      | 0.97   | 0.97     | 28976   |
+| **Macro avg** | 0.97      | 0.96   | 0.97     | 28976   |
+| **Weighted avg** | 0.98   | 0.97   | 0.97     | 28976   |
+
+
+## A list of entities that the model is able to recognize.
+
+```python
+[
+    "AGE",
+    "BUILDINGNUM",
+    "CITY",
+    "CREDITCARDNUMBER",
+    "DATE",
+    "DRIVERLICENSENUM",
+    "EMAIL",
+    "GIVENNAME",
+    "IDCARDNUM",
+    "PASSPORTNUM",
+    "STREET",
+    "SURNAME",
+    "TAXNUM",
+    "TELEPHONENUM",
+    "TIME",
+    "ZIPCODE"
+]
+
+```
+
+## Usage
+
+To use the model for spell correction:
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForTokenClassification
+
+model_id = "LocalDoc/private_ner_azerbaijani_v2"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForTokenClassification.from_pretrained(model_id)
+
+test_text = (
+    "Salam, mənim adım Əli Hüseynovdur. Doğum tarixim 15.05.1990-dır. Bakı şəhərində, Nizami küçəsində, 25/31 ünvanında yaşayıram. Telefon nömrəm +994552345678-dir."
+)
+
+inputs = tokenizer(test_text, return_tensors="pt", return_offsets_mapping=True)
+
+offset_mapping = inputs.pop("offset_mapping")
+
+with torch.no_grad():
+    outputs = model(**inputs)
+
+predictions = torch.argmax(outputs.logits, dim=2)
+
+tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
+offset_mapping = offset_mapping[0].tolist()
+predicted_labels = [model.config.id2label[pred.item()] for pred in predictions[0]]
+word_ids = inputs.word_ids(batch_index=0)
+
+aggregated = []
+prev_word_id = None
+for idx, word_id in enumerate(word_ids):
+    if word_id is None:
+        continue
+    if word_id != prev_word_id:
+        aggregated.append({
+            "word_id": word_id,
+            "tokens": [tokens[idx]],
+            "offsets": [offset_mapping[idx]],
+            "label": predicted_labels[idx]
+        })
+    else:
+        aggregated[-1]["tokens"].append(tokens[idx])
+        aggregated[-1]["offsets"].append(offset_mapping[idx])
+    prev_word_id = word_id
+
+entities = []
+current_entity = None
+for word in aggregated:
+    if word["label"] == "O":
+        if current_entity is not None:
+            entities.append(current_entity)
+            current_entity = None
+    else:
+        if current_entity is None:
+            current_entity = {
+                "type": word["label"],
+                "start": word["offsets"][0][0],
+                "end": word["offsets"][-1][1]
+            }
+        else:
+            if word["label"] == current_entity["type"]:
+                current_entity["end"] = word["offsets"][-1][1]
+            else:
+                entities.append(current_entity)
+                current_entity = {
+                    "type": word["label"],
+                    "start": word["offsets"][0][0],
+                    "end": word["offsets"][-1][1]
+                }
+if current_entity is not None:
+    entities.append(current_entity)
+
+for entity in entities:
+    entity["text"] = test_text[entity["start"]:entity["end"]]
+
+for entity in entities:
+    print(entity)
+```
+
+```json
+{'type': 'FIRSTNAME', 'start': 18, 'end': 21, 'text': 'Əli'}
+{'type': 'LASTNAME', 'start': 22, 'end': 34, 'text': 'Hüseynovdur.'}
+{'type': 'DOB', 'start': 49, 'end': 64, 'text': '15.05.1990-dır.'}
+{'type': 'STREET', 'start': 81, 'end': 87, 'text': 'Nizami'}
+{'type': 'BUILDINGNUMBER', 'start': 99, 'end': 104, 'text': '25/31'}
+{'type': 'PHONENUMBER', 'start': 141, 'end': 159, 'text': '+994552345678-dir.'}
+```
+
+
+## CC BY 4.0 License — What It Allows
+
+The **Creative Commons Attribution 4.0 International (CC BY 4.0)** license allows:
+
+### ✅ You Can:
+- **Use** the model for any purpose, including commercial use.
+- **Share** it — copy and redistribute in any medium or format.
+- **Adapt** it — remix, transform, and build upon it for any purpose, even commercially.
+
+### 📝 You Must:
+- **Give appropriate credit** — Attribute the original creator (e.g., name, link to the license, and indicate if changes were made).
+- **Not imply endorsement** — Do not suggest the original author endorses you or your use.
+
+### ❌ You Cannot:
+- Apply legal terms or technological measures that legally restrict others from doing anything the license permits (no DRM or additional restrictions).
+
+
+### Summary:
+You are free to use, modify, and distribute the model — even for commercial purposes — as long as you give proper credit to the original creator.
+
+
+For more information, please refer to the <a target="_blank" href="https://creativecommons.org/licenses/by/4.0/deed.en">CC BY-NC-ND 4.0 license</a>.
+
+
+## Contact
+
+For more information, questions, or issues, please contact LocalDoc at [[email protected]].