Hugging Face's logo

deb101
/
mistral-7b-instruct-v0.3-mimic4-adapt-l2r

Transformers
Safetensors
ltr_model
Generated from Trainer
4-bit precision
bitsandbytes
Model card Files
xet
Community
deb101 commited on
Commit
c0a12f0
·
verified ·
1 Parent(s): 4e4441b

Model save

Browse files
Files changed (10) hide show
  1. README.md +5 -5
  2. eval_loss_plot.png +0 -0
  3. eval_ndcg@25_plot.png +0 -0
  4. ground_model/README.md +202 -0
  5. ground_model/adapter_config.json +36 -0
  6. ground_model/adapter_model.safetensors +3 -0
  7. special_tokens_map.json +7 -1
  8. train_loss_plot.png +0 -0
  9. training_args.bin +1 -1
  10. training_l2r_log_2025-06-11_16-01-58.log +506 -0
README.md CHANGED
@@ -16,10 +16,10 @@ should probably proofread and complete it, then remove this comment. -->
16
 
17
  This model is a fine-tuned version of [mistralai/Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) on the None dataset.
18
  It achieves the following results on the evaluation set:
19
- - Loss: -377379303853489216.0000
20
- - Ndcg: 0.9566
21
- - Ndcg@25: 0.5389
22
- - Precision@25: 0.2423
23
 
24
  ## Model description
25
 
@@ -54,7 +54,7 @@ The following hyperparameters were used during training:
54
  | Training Loss | Epoch | Step | Validation Loss | Ndcg | Ndcg@25 | Precision@25 |
55
  |:-------------------------:|:------:|:----:|:------------------------:|:------:|:-------:|:------------:|
56
  | -6819856770917832704.0000 | 1.0 | 44 | -289695770057323328.0000 | 0.9560 | 0.6207 | 0.2464 |
57
- | -160735776926492256.0000 | 1.9711 | 86 | -377379303853489216.0000 | 0.9566 | 0.5389 | 0.2423 |
58
 
59
 
60
  ### Framework versions
 
16
 
17
  This model is a fine-tuned version of [mistralai/Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) on the None dataset.
18
  It achieves the following results on the evaluation set:
19
+ - Loss: -392272726550942272.0000
20
+ - Ndcg: 0.9564
21
+ - Ndcg@25: 0.5959
22
+ - Precision@25: 0.2234
23
 
24
  ## Model description
25
 
 
54
  | Training Loss | Epoch | Step | Validation Loss | Ndcg | Ndcg@25 | Precision@25 |
55
  |:-------------------------:|:------:|:----:|:------------------------:|:------:|:-------:|:------------:|
56
  | -6819856770917832704.0000 | 1.0 | 44 | -289695770057323328.0000 | 0.9560 | 0.6207 | 0.2464 |
57
+ | -164647509444630944.0000 | 1.9711 | 86 | -392272726550942272.0000 | 0.9564 | 0.5959 | 0.2234 |
58
 
59
 
60
  ### Framework versions
eval_loss_plot.png CHANGED
eval_ndcg@25_plot.png CHANGED
ground_model/README.md ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: mistralai/Mistral-7B-Instruct-v0.3
3
+ library_name: peft
4
+ ---
5
+
6
+ # Model Card for Model ID
7
+
8
+ <!-- Provide a quick summary of what the model is/does. -->
9
+
10
+
11
+
12
+ ## Model Details
13
+
14
+ ### Model Description
15
+
16
+ <!-- Provide a longer summary of what this model is. -->
17
+
18
+
19
+
20
+ - **Developed by:** [More Information Needed]
21
+ - **Funded by [optional]:** [More Information Needed]
22
+ - **Shared by [optional]:** [More Information Needed]
23
+ - **Model type:** [More Information Needed]
24
+ - **Language(s) (NLP):** [More Information Needed]
25
+ - **License:** [More Information Needed]
26
+ - **Finetuned from model [optional]:** [More Information Needed]
27
+
28
+ ### Model Sources [optional]
29
+
30
+ <!-- Provide the basic links for the model. -->
31
+
32
+ - **Repository:** [More Information Needed]
33
+ - **Paper [optional]:** [More Information Needed]
34
+ - **Demo [optional]:** [More Information Needed]
35
+
36
+ ## Uses
37
+
38
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
39
+
40
+ ### Direct Use
41
+
42
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
43
+
44
+ [More Information Needed]
45
+
46
+ ### Downstream Use [optional]
47
+
48
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
49
+
50
+ [More Information Needed]
51
+
52
+ ### Out-of-Scope Use
53
+
54
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
55
+
56
+ [More Information Needed]
57
+
58
+ ## Bias, Risks, and Limitations
59
+
60
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
61
+
62
+ [More Information Needed]
63
+
64
+ ### Recommendations
65
+
66
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
67
+
68
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
69
+
70
+ ## How to Get Started with the Model
71
+
72
+ Use the code below to get started with the model.
73
+
74
+ [More Information Needed]
75
+
76
+ ## Training Details
77
+
78
+ ### Training Data
79
+
80
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
81
+
82
+ [More Information Needed]
83
+
84
+ ### Training Procedure
85
+
86
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
87
+
88
+ #### Preprocessing [optional]
89
+
90
+ [More Information Needed]
91
+
92
+
93
+ #### Training Hyperparameters
94
+
95
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
96
+
97
+ #### Speeds, Sizes, Times [optional]
98
+
99
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
100
+
101
+ [More Information Needed]
102
+
103
+ ## Evaluation
104
+
105
+ <!-- This section describes the evaluation protocols and provides the results. -->
106
+
107
+ ### Testing Data, Factors & Metrics
108
+
109
+ #### Testing Data
110
+
111
+ <!-- This should link to a Dataset Card if possible. -->
112
+
113
+ [More Information Needed]
114
+
115
+ #### Factors
116
+
117
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
118
+
119
+ [More Information Needed]
120
+
121
+ #### Metrics
122
+
123
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
124
+
125
+ [More Information Needed]
126
+
127
+ ### Results
128
+
129
+ [More Information Needed]
130
+
131
+ #### Summary
132
+
133
+
134
+
135
+ ## Model Examination [optional]
136
+
137
+ <!-- Relevant interpretability work for the model goes here -->
138
+
139
+ [More Information Needed]
140
+
141
+ ## Environmental Impact
142
+
143
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
144
+
145
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
146
+
147
+ - **Hardware Type:** [More Information Needed]
148
+ - **Hours used:** [More Information Needed]
149
+ - **Cloud Provider:** [More Information Needed]
150
+ - **Compute Region:** [More Information Needed]
151
+ - **Carbon Emitted:** [More Information Needed]
152
+
153
+ ## Technical Specifications [optional]
154
+
155
+ ### Model Architecture and Objective
156
+
157
+ [More Information Needed]
158
+
159
+ ### Compute Infrastructure
160
+
161
+ [More Information Needed]
162
+
163
+ #### Hardware
164
+
165
+ [More Information Needed]
166
+
167
+ #### Software
168
+
169
+ [More Information Needed]
170
+
171
+ ## Citation [optional]
172
+
173
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
174
+
175
+ **BibTeX:**
176
+
177
+ [More Information Needed]
178
+
179
+ **APA:**
180
+
181
+ [More Information Needed]
182
+
183
+ ## Glossary [optional]
184
+
185
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
186
+
187
+ [More Information Needed]
188
+
189
+ ## More Information [optional]
190
+
191
+ [More Information Needed]
192
+
193
+ ## Model Card Authors [optional]
194
+
195
+ [More Information Needed]
196
+
197
+ ## Model Card Contact
198
+
199
+ [More Information Needed]
200
+ ### Framework versions
201
+
202
+ - PEFT 0.15.2
ground_model/adapter_config.json ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alpha_pattern": {},
3
+ "auto_mapping": null,
4
+ "base_model_name_or_path": "mistralai/Mistral-7B-Instruct-v0.3",
5
+ "bias": "none",
6
+ "corda_config": null,
7
+ "eva_config": null,
8
+ "exclude_modules": null,
9
+ "fan_in_fan_out": false,
10
+ "inference_mode": true,
11
+ "init_lora_weights": true,
12
+ "layer_replication": null,
13
+ "layers_pattern": null,
14
+ "layers_to_transform": null,
15
+ "loftq_config": {},
16
+ "lora_alpha": 32,
17
+ "lora_bias": false,
18
+ "lora_dropout": 0.05,
19
+ "megatron_config": null,
20
+ "megatron_core": "megatron.core",
21
+ "modules_to_save": null,
22
+ "peft_type": "LORA",
23
+ "r": 16,
24
+ "rank_pattern": {},
25
+ "revision": null,
26
+ "target_modules": [
27
+ "k_proj",
28
+ "o_proj",
29
+ "q_proj",
30
+ "v_proj"
31
+ ],
32
+ "task_type": "CAUSAL_LM",
33
+ "trainable_token_indices": null,
34
+ "use_dora": false,
35
+ "use_rslora": false
36
+ }
ground_model/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:875ad4d3f6fb36ceeb05e9ee0e3f231999529696b97ae44b1607ad0ac4991753
3
+ size 54560368
special_tokens_map.json CHANGED
@@ -13,7 +13,13 @@
13
  "rstrip": false,
14
  "single_word": false
15
  },
16
- "pad_token": "</s>",
 
 
 
 
 
 
17
  "unk_token": {
18
  "content": "<unk>",
19
  "lstrip": false,
 
13
  "rstrip": false,
14
  "single_word": false
15
  },
16
+ "pad_token": {
17
+ "content": "</s>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
  "unk_token": {
24
  "content": "<unk>",
25
  "lstrip": false,
train_loss_plot.png CHANGED
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:9d8ecf06721ad87ab8121f2db0c7f9e06711bbd37d7a94b1f2e0165aeb84b036
3
  size 5432
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:32ddf0a3de16457bc32a43966b2bfde212cfda1c4ba6171aa8231284c3276345
3
  size 5432
training_l2r_log_2025-06-11_16-01-58.log ADDED
@@ -0,0 +1,506 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2025-06-11 16:01:58,980 - INFO - 📝 Logging initialized. Log file created at: ../tmp/logs/training_l2r_log_2025-06-11_16-01-58.log - [learning2rank.py:287:setup_logger]
2
+ 2025-06-11 16:01:58,980 - INFO - ================================================================================ - [learning2rank.py:109:log_section]
3
+ 2025-06-11 16:01:58,980 - INFO - = 📌 INITIALIZING TRAINING ENVIRONMENT = - [learning2rank.py:110:log_section]
4
+ 2025-06-11 16:01:58,980 - INFO - ================================================================================ - [learning2rank.py:113:log_section]
5
+ 2025-06-11 16:01:58,980 - INFO - 🚀 Setting up data paths and environment variables... - [learning2rank.py:3766:main]
6
+ 2025-06-11 16:01:58,981 - INFO - 🛠️ Command-line Arguments: - [learning2rank.py:303:print_args]
7
+ 2025-06-11 16:01:58,981 - INFO -
8
+ 🔹 output_dir: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b
9
+ 🔹 source_url: XURLs.MIMIC4_DEMO
10
+ 🔹 data: mimic4_icd10_full
11
+ 🔹 data_l2r_fname_prefix: mimic4_icd10
12
+ 🔹 logfile: training_l2r_log
13
+ 🔹 base_dir: ../tmp/MIMIC4_DEMO
14
+ 🔹 l2r_boot_dir: ../tmp/MIMIC4_DEMO/mimic4_l2rboot_mistral7b
15
+ 🔹 hub_model_id: deb101/mistral-7b-instruct-v0.3-mimic4-adapt
16
+ 🔹 model_name: mistralai/Mistral-7B-Instruct-v0.3
17
+ 🔹 max_length: 512
18
+ 🔹 do_fresh_training: True
19
+ 🔹 load_from_checkpoint: True
20
+ 🔹 task: l2r
21
+ 🔹 num_train_epochs: 2
22
+ 🔹 metric_for_best_model: ndcg@25
23
+ 🔹 learning_rate: 0.0001
24
+ 🔹 warmup_steps: 0
25
+ 🔹 generate_report: False
26
+ 🔹 logfile_path: ../tmp/logs/training_l2r_log_2025-06-11_16-01-58.log
27
+ 🔹 source: /home/ubuntu/.xcube/data/mimic4_demo - [learning2rank.py:304:print_args]
28
+ 2025-06-11 16:01:58,981 - INFO - ➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖ - [learning2rank.py:305:print_args]
29
+ 2025-06-11 16:01:58,981 - INFO - 📁 Using L2R boot directory: ../tmp/MIMIC4_DEMO/mimic4_l2rboot_mistral7b - [learning2rank.py:3780:main]
30
+ 2025-06-11 16:01:58,981 - INFO - 📂 Using output directory: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b - [learning2rank.py:3782:main]
31
+ 2025-06-11 16:01:58,981 - INFO - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - [learning2rank.py:109:log_section]
32
+ 2025-06-11 16:01:58,981 - INFO - + ✨ LOADING DATASETS + - [learning2rank.py:110:log_section]
33
+ 2025-06-11 16:01:58,981 - INFO - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - [learning2rank.py:113:log_section]
34
+ 2025-06-11 16:01:58,981 - INFO - 📊 Loading main dataset and L2R dataset... - [learning2rank.py:3785:main]
35
+ 2025-06-11 16:01:58,981 - INFO - 📂 Loading main data from: /home/ubuntu/.xcube/data/mimic4_demo/mimic4_icd10_full.csv - [learning2rank.py:333:get_data]
36
+ 2025-06-11 16:02:07,226 - INFO - ✅ Successfully loaded main data: 122279 rows - [learning2rank.py:346:get_data]
37
+ 2025-06-11 16:02:07,226 - INFO - 📂 Loading L2R data from: ../tmp/MIMIC4_DEMO/mimic4_l2rboot_mistral7b/mimic4_icd10_tok_rank_per_lbl.ft - [learning2rank.py:352:get_data]
38
+ 2025-06-11 16:02:07,226 - INFO - 📂 Loading L2R tokens from: ../tmp/MIMIC4_DEMO/mimic4_l2rboot_mistral7b/mimic4_icd10_tok.ft - [learning2rank.py:355:get_data]
39
+ 2025-06-11 16:02:07,226 - INFO - 📂 Loading L2R labels from: ../tmp/MIMIC4_DEMO/mimic4_l2rboot_mistral7b/mimic4_icd10_lbl.ft - [learning2rank.py:358:get_data]
40
+ 2025-06-11 16:02:08,933 - INFO - ✅ Successfully loaded L2R data: 260243456 rows - [learning2rank.py:365:get_data]
41
+ 2025-06-11 16:02:08,934 - INFO - ✅ Successfully loaded L2R tokens: 32768 tokens - [learning2rank.py:366:get_data]
42
+ 2025-06-11 16:02:08,934 - INFO - ✅ Successfully loaded L2R labels: 7942 rows - [learning2rank.py:367:get_data]
43
+ 2025-06-11 16:02:08,934 - INFO - 🔄 Total data loaded: 122279 main rows, 260243456 L2R rows - [learning2rank.py:374:get_data]
44
+ 2025-06-11 16:02:08,934 - INFO - ✨ Successfully loaded both datasets: - [learning2rank.py:398:load_datasets]
45
+ 2025-06-11 16:02:08,934 - INFO - - 📄 Main dataset: 122279 records - [learning2rank.py:399:load_datasets]
46
+ 2025-06-11 16:02:08,934 - INFO - - 📄 L2R dataset: 260243456 records - [learning2rank.py:400:load_datasets]
47
+ 2025-06-11 16:02:08,934 - INFO - - 🔤 Tokens: 32768 items - [learning2rank.py:401:load_datasets]
48
+ 2025-06-11 16:02:08,934 - INFO - - 🏷️ Labels: 7942 items - [learning2rank.py:402:load_datasets]
49
+ 2025-06-11 16:02:08,941 - INFO - ✅ Data loading completed successfully - [learning2rank.py:410:load_datasets]
50
+ 2025-06-11 16:02:10,246 - INFO - ⏳ Starting quantization of ranks for DataFrame with 260243456 rows. Containing 32768 unique tokens & 7942 unique labels - [learning2rank.py:529:quantize_l2r_data]
51
+ 2025-06-11 16:02:10,247 - INFO - 🔄 Quantizing those 32768 unique token ranks into 101 quantization levels for each label - [learning2rank.py:554:quantize_l2r_data]
52
+ 2025-06-11 16:02:57,083 - INFO - ✅ Completed quantization: Produced tensor of shape torch.Size([7942, 32768, 4]) with 101 quantization levels per label - [learning2rank.py:608:quantize_l2r_data]
53
+ 2025-06-11 16:02:57,110 - WARNING - Label 2700: Only 1 tokens with top relevance score (need 50) - [learning2rank.py:724:test_scored_tokens]
54
+ 2025-06-11 16:02:57,115 - WARNING - Label 2808: Only 1 tokens with top relevance score (need 50) - [learning2rank.py:724:test_scored_tokens]
55
+ 2025-06-11 16:02:57,119 - WARNING - Label 7568: Only 1 tokens with top relevance score (need 50) - [learning2rank.py:724:test_scored_tokens]
56
+ 2025-06-11 16:02:57,123 - WARNING - Label 3360: Only 1 tokens with top relevance score (need 50) - [learning2rank.py:724:test_scored_tokens]
57
+ 2025-06-11 16:02:57,127 - WARNING - Label 793: Only 1 tokens with top relevance score (need 50) - [learning2rank.py:724:test_scored_tokens]
58
+ 2025-06-11 16:02:57,155 - INFO - ******************************************************************************** - [learning2rank.py:109:log_section]
59
+ 2025-06-11 16:02:57,155 - INFO - * 🌟 STARTING LEARNING TO RANK MODEL TRAINING * - [learning2rank.py:110:log_section]
60
+ 2025-06-11 16:02:57,155 - INFO - ******************************************************************************** - [learning2rank.py:113:log_section]
61
+ 2025-06-11 16:02:57,155 - INFO - 🔐 Loaded authentication token from environment - [learning2rank.py:3814:main]
62
+ 2025-06-11 16:02:57,155 - INFO - 🏷️ Hub Model ID for this Learning to Rank task: deb101/mistral-7b-instruct-v0.3-mimic4-adapt-l2r - [learning2rank.py:3818:main]
63
+ 2025-06-11 16:02:57,155 - INFO - -------------------------------------------------------------------------------- - [learning2rank.py:109:log_section]
64
+ 2025-06-11 16:02:57,155 - INFO - - 📋 MODEL EXISTENCE CHECK - - [learning2rank.py:110:log_section]
65
+ 2025-06-11 16:02:57,155 - INFO - -------------------------------------------------------------------------------- - [learning2rank.py:113:log_section]
66
+ 2025-06-11 16:02:57,156 - INFO - 🔍 Checking model existence locally and on Hugging Face Hub... - [learning2rank.py:430:check_model_existence]
67
+ 2025-06-11 16:02:57,156 - INFO - ✅ Model exists locally at: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b - [learning2rank.py:435:check_model_existence]
68
+ 2025-06-11 16:02:57,217 - INFO - ✅ Model exists on Hugging Face Hub with ID: deb101/mistral-7b-instruct-v0.3-mimic4-adapt-l2r - [learning2rank.py:449:check_model_existence]
69
+ 2025-06-11 16:02:57,218 - INFO - 📁 Model exists either locally or on Hub - [learning2rank.py:475:check_model_existence]
70
+ 2025-06-11 16:02:57,218 - INFO - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - [learning2rank.py:109:log_section]
71
+ 2025-06-11 16:02:57,218 - INFO - + ✨ STARTING FRESH TRAINING + - [learning2rank.py:110:log_section]
72
+ 2025-06-11 16:02:57,218 - INFO - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - [learning2rank.py:113:log_section]
73
+ 2025-06-11 16:02:57,218 - INFO - 🔄 Starting fresh training (either forced or model not found)... - [learning2rank.py:3831:main]
74
+ 2025-06-11 16:02:57,231 - WARNING - Note: Environment variable`HF_TOKEN` is set and is the current active token independently from the token you've just configured. - [_login.py:415:_login]
75
+ 2025-06-11 16:02:57,232 - INFO - 🔑 Successfully authenticated with Hugging Face Hub - [learning2rank.py:311:authenticate_hf]
76
+ 2025-06-11 16:02:57,232 - INFO - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - [learning2rank.py:109:log_section]
77
+ 2025-06-11 16:02:57,232 - INFO - + ✨ DATA PREPROCESSING + - [learning2rank.py:110:log_section]
78
+ 2025-06-11 16:02:57,232 - INFO - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - [learning2rank.py:113:log_section]
79
+ 2025-06-11 16:02:57,232 - INFO - 🔄 Loading and preprocessing training data... - [learning2rank.py:3838:main]
80
+ 2025-06-11 16:02:57,232 - INFO - 🔤 Loading tokenizer for model: deb101/mistral-7b-instruct-v0.3-mimic4-adapt-l2r... - [learning2rank.py:3710:load_tokenizer]
81
+ 2025-06-11 16:02:57,258 - INFO - 📅 Repository last modified: 2025-06-11 15:41:14+00:00 - [learning2rank.py:3718:load_tokenizer]
82
+ 2025-06-11 16:02:58,254 - INFO - ✅ Tokenizer loaded successfully for model: deb101/mistral-7b-instruct-v0.3-mimic4-adapt-l2r - [learning2rank.py:3729:load_tokenizer]
83
+ 2025-06-11 16:02:59,524 - INFO - 🔍 Verifying uniqueness of token IDs per label in scored_tokens... - [learning2rank.py:1302:preprocess_data]
84
+ 2025-06-11 16:03:00,361 - INFO - ✅ All labels have unique token IDs in scored_tokens. - [learning2rank.py:1314:preprocess_data]
85
+ 2025-06-11 16:03:00,361 - INFO - 🚀 Building a 2D lookup table for efficient token-to-relevance mapping across all labels! 🚀 - [learning2rank.py:1317:preprocess_data]
86
+ 2025-06-11 16:03:00,361 - INFO - 🔢 Total labels = 7942 - [learning2rank.py:1320:preprocess_data]
87
+ 2025-06-11 16:03:00,361 - INFO - 🔍 Precomputing token indices and corresponding relevance_values for each label... - [learning2rank.py:1321:preprocess_data]
88
+ 2025-06-11 16:03:00,590 - INFO - 📊 Lookup table dimensions: 32768 vocabulary size × 7942 labels - [learning2rank.py:1328:preprocess_data]
89
+ 2025-06-11 16:03:00,590 - INFO - ⚡ This approach eliminates token comparison broadcasting and provides O(1) lookup time for relevance scores! - [learning2rank.py:1331:preprocess_data]
90
+ 2025-06-11 16:03:00,590 - INFO - 🧮 Processing relevance calculations vectorized for maximum speed 🔥 - [learning2rank.py:1334:preprocess_data]
91
+ 2025-06-11 16:03:00,759 - INFO - 🔍 Verifying token mappings with 10 samples... - [learning2rank.py:1367:verify_token_mappings]
92
+ 2025-06-11 16:03:00,949 - INFO - ✅ Token mappings verification completed successfully! 🎉 - [learning2rank.py:1458:verify_token_mappings]
93
+ 2025-06-11 16:03:00,953 - INFO - 🔄 Processing dataset with map... - [learning2rank.py:1522:preprocess_data]
94
+ 2025-06-11 16:03:01,271 - INFO - ✅ Dataset built in 0h 0m 0.32s - [learning2rank.py:1545:preprocess_data]
95
+ 2025-06-11 16:03:01,284 - INFO - The size of Training set: 173 🏋️ (Training Data) - [learning2rank.py:1576:preprocess_data]
96
+ 2025-06-11 16:03:01,284 - INFO - The size of Evaluation set: 35 🔍 (Test Data) - [learning2rank.py:1577:preprocess_data]
97
+ 2025-06-11 16:03:01,284 - INFO - 🚀 Created HuggingFace Dataset with 208 samples, 7942 labels - [learning2rank.py:1585:preprocess_data]
98
+ 2025-06-11 16:03:01,285 - INFO - 🏷️ Number of unique ICD-10 codes: 7942 - [learning2rank.py:3852:main]
99
+ 2025-06-11 16:03:01,285 - INFO - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - [learning2rank.py:109:log_section]
100
+ 2025-06-11 16:03:01,285 - INFO - + ✨ LOADING FROM CHECKPOINT + - [learning2rank.py:110:log_section]
101
+ 2025-06-11 16:03:01,285 - INFO - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - [learning2rank.py:113:log_section]
102
+ 2025-06-11 16:03:01,285 - INFO - 📥 Loading existing model from checkpoint... - [learning2rank.py:3855:main]
103
+ 2025-06-11 16:03:01,285 - INFO - 🔍 Model exists locally: True - [learning2rank.py:3511:load_custom_task_model_and_tokenizer]
104
+ 2025-06-11 16:03:01,285 - INFO - 🌐 Loading model from Hub: deb101/mistral-7b-instruct-v0.3-mimic4-adapt-l2r - [learning2rank.py:3519:load_custom_task_model_and_tokenizer]
105
+ 2025-06-11 16:03:01,389 - INFO - 🌐 Loaded config from Hub: deb101/mistral-7b-instruct-v0.3-mimic4-adapt-l2r - [learning2rank.py:3537:load_custom_task_model_and_tokenizer]
106
+ 2025-06-11 16:03:01,389 - INFO - 🔢 Num labels: 7942, Base model path: mistralai/Mistral-7B-Instruct-v0.3 - [learning2rank.py:3551:load_custom_task_model_and_tokenizer]
107
+ 2025-06-11 16:03:01,389 - INFO - 🚀 Starting model and tokenizer loading process... - [learning2rank.py:939:load_base_model_and_tokenizer]
108
+ 2025-06-11 16:03:01,390 - INFO - 📊 Quantization config: BitsAndBytesConfig {
109
+ "_load_in_4bit": true,
110
+ "_load_in_8bit": false,
111
+ "bnb_4bit_compute_dtype": "bfloat16",
112
+ "bnb_4bit_quant_storage": "uint8",
113
+ "bnb_4bit_quant_type": "nf4",
114
+ "bnb_4bit_use_double_quant": true,
115
+ "llm_int8_enable_fp32_cpu_offload": false,
116
+ "llm_int8_has_fp16_weight": false,
117
+ "llm_int8_skip_modules": null,
118
+ "llm_int8_threshold": 6.0,
119
+ "load_in_4bit": true,
120
+ "load_in_8bit": false,
121
+ "quant_method": "bitsandbytes"
122
+ }
123
+ - [learning2rank.py:948:load_base_model_and_tokenizer]
124
+ 2025-06-11 16:03:01,390 - INFO - 🔤 Loading tokenizer for model: mistralai/Mistral-7B-Instruct-v0.3... - [learning2rank.py:952:load_base_model_and_tokenizer]
125
+ 2025-06-11 16:03:01,695 - INFO - 📝 Setting pad token to eos token... - [learning2rank.py:956:load_base_model_and_tokenizer]
126
+ 2025-06-11 16:03:01,695 - INFO - 🧠 Loading base model with quantization... - [learning2rank.py:964:load_base_model_and_tokenizer]
127
+ 2025-06-11 16:03:02,001 - INFO - We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk). - [modeling.py:991:get_balanced_memory]
128
+ 2025-06-11 16:03:06,689 - INFO - 🔧 Setting up default LoRA configuration... - [learning2rank.py:987:load_base_model_and_tokenizer]
129
+ 2025-06-11 16:03:06,689 - INFO - 🔍 LoRA config: r=16, alpha=32, targets={'k_proj', 'o_proj', 'q_proj', 'v_proj'}, dropout=0.05 - [learning2rank.py:1010:load_base_model_and_tokenizer]
130
+ 2025-06-11 16:03:06,689 - INFO - 🧩 Applying LoRA adapters to model... - [learning2rank.py:1017:load_base_model_and_tokenizer]
131
+ 2025-06-11 16:03:06,872 - INFO - 📊 trainable params: 13,631,488 || all params: 7,261,655,040 || trainable%: 0.1877 - [learning2rank.py:135:log_print_output]
132
+ 2025-06-11 16:03:06,872 - INFO - ✅ Model and tokenizer successfully loaded! - [learning2rank.py:1024:load_base_model_and_tokenizer]
133
+ 2025-06-11 16:03:06,872 - INFO - 🤖 Loaded base model from: mistralai/Mistral-7B-Instruct-v0.3 - [learning2rank.py:3555:load_custom_task_model_and_tokenizer]
134
+ 2025-06-11 16:03:06,872 - INFO - Before preparing the model the % of trainable params are: - [learning2rank.py:3563:load_custom_task_model_and_tokenizer]
135
+ 2025-06-11 16:03:06,876 - INFO - 📊 trainable params: 13,631,488 || all params: 3,771,994,112 || trainable%: 0.3614 - [learning2rank.py:1699:compute_trainable_params]
136
+ 2025-06-11 16:03:08,428 - INFO - 🔧 Registering LTRModel with transformers AutoModel 🚀 - [learning2rank.py:1725:define_model]
137
+ 2025-06-11 16:03:08,428 - INFO - 🏗️ Initialized L2R model with 7942 labels - [learning2rank.py:3566:load_custom_task_model_and_tokenizer]
138
+ 2025-06-11 16:03:08,428 - INFO - After preparing the model the % of trainable params are: - [learning2rank.py:3567:load_custom_task_model_and_tokenizer]
139
+ 2025-06-11 16:03:08,433 - INFO - 📊 trainable params: 46,161,920 || all params: 3,804,524,544 || trainable%: 1.2133 - [core.py:293:compute_trainable_params]
140
+ 2025-06-11 16:03:08,455 - INFO - 🖥️ Model moved to device: cuda - [learning2rank.py:3572:load_custom_task_model_and_tokenizer]
141
+ 2025-06-11 16:04:30,575 - INFO - 💾 Loaded custom weights from Hub: /home/ubuntu/.cache/huggingface/hub/models--deb101--mistral-7b-instruct-v0.3-mimic4-adapt-l2r/snapshots/1e76e17d2d57f42090ec09e039a2e2ae47dc1ec1/model.safetensors - [learning2rank.py:3595:load_custom_task_model_and_tokenizer]
142
+ 2025-06-11 16:04:31,642 - INFO - ✅ Custom weights loaded into model - [learning2rank.py:3614:load_custom_task_model_and_tokenizer]
143
+ 2025-06-11 16:04:31,663 - INFO - ✔️ Loaded weights match model's state_dict - [learning2rank.py:3625:load_custom_task_model_and_tokenizer]
144
+ 2025-06-11 16:04:31,663 - INFO - 📋 Loaded files from Hub (directory: /home/ubuntu/.cache/huggingface/hub/models--deb101--mistral-7b-instruct-v0.3-mimic4-adapt-l2r/snapshots/1e76e17d2d57f42090ec09e039a2e2ae47dc1ec1):
145
+ +---------+-------------------+------------+
146
+ | Index | Loaded File | Size |
147
+ +=========+===================+============+
148
+ | 1 | model.safetensors | 4122.74 MB |
149
+ +---------+-------------------+------------+
150
+ | 2 | config.json | 0.38 MB |
151
+ +---------+-------------------+------------+ - [learning2rank.py:3665:load_custom_task_model_and_tokenizer]
152
+ 2025-06-11 16:04:32,136 - INFO - 🖌️ Loaded tokenizer from: deb101/mistral-7b-instruct-v0.3-mimic4-adapt-l2r - [learning2rank.py:3680:load_custom_task_model_and_tokenizer]
153
+ 2025-06-11 16:04:32,147 - INFO - 🖥️ Model moved to device: cuda - [learning2rank.py:3687:load_custom_task_model_and_tokenizer]
154
+ 2025-06-11 16:04:32,159 - INFO - ✅ Successfully loaded from checkpoint, will now continue training for 2 epochs. - [learning2rank.py:3866:main]
155
+ 2025-06-11 16:04:32,162 - INFO - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - [learning2rank.py:109:log_section]
156
+ 2025-06-11 16:04:32,163 - INFO - + ✨ TRAINING PREPARATION + - [learning2rank.py:110:log_section]
157
+ 2025-06-11 16:04:32,163 - INFO - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - [learning2rank.py:113:log_section]
158
+ 2025-06-11 16:04:32,163 - INFO - ⚙️ Preparing training components and optimizers... - [learning2rank.py:3904:main]
159
+ 2025-06-11 16:04:32,193 - INFO - 🖥️ Device: NVIDIA GH200 480GB - [learning2rank.py:1887:log_training_configuration]
160
+ 2025-06-11 16:04:32,193 - INFO - 🔋 CUDA Available: True - [learning2rank.py:1890:log_training_configuration]
161
+ 2025-06-11 16:04:32,193 - INFO - 💾 CUDA Device Count: 1 - [learning2rank.py:1891:log_training_configuration]
162
+ 2025-06-11 16:04:32,194 - INFO -
163
+ 📋 Training Configuration 📋
164
+ +----------+-----------------------------+--------------------------------------------------+
165
+ | 🌟 Emoji | 🏷️ Parameter | 📊 Value |
166
+ +----------+-----------------------------+--------------------------------------------------+
167
+ | 📁 | Output Directory | ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b |
168
+ | 🔁 | Training Epochs | 2 |
169
+ | 🏋️ | Train Batch Size | 1 |
170
+ | 🔍 | Eval Batch Size | 1 |
171
+ | 📊 | Gradient Accumulation Steps | 4 |
172
+ | 🚀 | Learning Rate | 0.0001 |
173
+ | 🌅 | Warmup Steps | 0 |
174
+ | 💾 | Save Strategy | epoch |
175
+ | 💾 | Save Total Limit | 10 |
176
+ | 📊 | Evaluation Strategy | epoch |
177
+ | 🎯 | Best Model Metric | ndcg@25 |
178
+ | 📝 | Logging Strategy | steps (every 10 steps) |
179
+ | 🌐 | Push to Hub | True |
180
+ | 🌐 | Hub Model ID | deb101/mistral-7b-instruct-v0.3-mimic4-adapt-l2r |
181
+ | 🔢 | Steps per Epoch | 43 |
182
+ | 🔢 | Total Training Steps | 86 |
183
+ | 🔢 | Evaluation Steps | 35 |
184
+ | 📊 | Training Dataset Size | 173 samples 🏋️ |
185
+ | 📊 | Evaluation Dataset Size | 35 samples 🔍 |
186
+ +----------+-----------------------------+--------------------------------------------------+ - [learning2rank.py:1879:log_training_args]
187
+ 2025-06-11 16:04:32,194 - INFO - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - [learning2rank.py:109:log_section]
188
+ 2025-06-11 16:04:32,194 - INFO - + ✨ MODEL TRAINING + - [learning2rank.py:110:log_section]
189
+ 2025-06-11 16:04:32,194 - INFO - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - [learning2rank.py:113:log_section]
190
+ 2025-06-11 16:04:32,195 - INFO - 🏋️ Starting model training process... - [learning2rank.py:3924:main]
191
+ 2025-06-11 16:04:32,195 - INFO - 🏁 Preparing Custom Trainer 🛠️ - [learning2rank.py:3080:train_model]
192
+ 2025-06-11 16:04:32,243 - INFO - We are registering the tokenizer deb101/mistral-7b-instruct-v0.3-mimic4-adapt-l2r in Custom Trainer - [learning2rank.py:2519:__init__]
193
+ 2025-06-11 16:04:32,243 - INFO - 🏋️ Commencing Model Training 💪 - [learning2rank.py:3121:train_model]
194
+ 2025-06-11 16:04:32,519 - INFO - 🚀 Starting Training... - [learning2rank.py:2269:on_train_begin]
195
+ 2025-06-11 16:04:51,857 - INFO -
196
+ 🚂 Training Metrics (Step 10) 🚂
197
+ +---------------+--------------+
198
+ | Metric | Value |
199
+ +===============+==============+
200
+ | loss | -1.33065e+17 |
201
+ +---------------+--------------+
202
+ | grad_norm | nan |
203
+ +---------------+--------------+
204
+ | learning_rate | 0.0001 |
205
+ +---------------+--------------+
206
+ | epoch | 0.231214 |
207
+ +---------------+--------------+ - [learning2rank.py:2187:on_log]
208
+ 2025-06-11 16:05:06,024 - INFO -
209
+ 🚂 Training Metrics (Step 20) 🚂
210
+ +---------------+--------------+
211
+ | Metric | Value |
212
+ +===============+==============+
213
+ | loss | -9.67589e+17 |
214
+ +---------------+--------------+
215
+ | grad_norm | nan |
216
+ +---------------+--------------+
217
+ | learning_rate | 0.0001 |
218
+ +---------------+--------------+
219
+ | epoch | 0.462428 |
220
+ +---------------+--------------+ - [learning2rank.py:2187:on_log]
221
+ 2025-06-11 16:05:20,233 - INFO -
222
+ 🚂 Training Metrics (Step 30) 🚂
223
+ +---------------+--------------+
224
+ | Metric | Value |
225
+ +===============+==============+
226
+ | loss | -1.80524e+17 |
227
+ +---------------+--------------+
228
+ | grad_norm | nan |
229
+ +---------------+--------------+
230
+ | learning_rate | 0.0001 |
231
+ +---------------+--------------+
232
+ | epoch | 0.693642 |
233
+ +---------------+--------------+ - [learning2rank.py:2187:on_log]
234
+ 2025-06-11 16:05:34,445 - INFO -
235
+ 🚂 Training Metrics (Step 40) 🚂
236
+ +---------------+--------------+
237
+ | Metric | Value |
238
+ +===============+==============+
239
+ | loss | -6.81986e+18 |
240
+ +---------------+--------------+
241
+ | grad_norm | nan |
242
+ +---------------+--------------+
243
+ | learning_rate | 0.0001 |
244
+ +---------------+--------------+
245
+ | epoch | 0.924855 |
246
+ +---------------+--------------+ - [learning2rank.py:2187:on_log]
247
+ 2025-06-11 16:05:39,045 - INFO - Removing 'token_type_ids' from eval_dataset as they are not needed. - [learning2rank.py:2635:evaluate]
248
+ 2025-06-11 16:08:54,739 - INFO -
249
+ 🔍 Evaluation Metrics 🔍
250
+ +-------------------+--------------+
251
+ | Metric | Value |
252
+ +===================+==============+
253
+ | eval_loss | -2.89696e+17 |
254
+ +-------------------+--------------+
255
+ | eval_ndcg | 0.956037 |
256
+ +-------------------+--------------+
257
+ | eval_ndcg@25 | 0.620746 |
258
+ +-------------------+--------------+
259
+ | eval_precision@25 | 0.2464 |
260
+ +-------------------+--------------+ - [learning2rank.py:2206:on_evaluate]
261
+ 2025-06-11 16:08:58,408 - INFO - 💾 Model weights saved in safetensors format: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/checkpoint-44 - [learning2rank.py:2759:_save]
262
+ 2025-06-11 16:08:58,435 - INFO - ⚙️ Config saved in checkpoint: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/checkpoint-44 - [learning2rank.py:2764:_save]
263
+ 2025-06-11 16:08:58,436 - INFO - 📋 Saved files in ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/checkpoint-44:
264
+ +---------+--------------------+------------+
265
+ | Index | Saved File | Size |
266
+ +=========+====================+============+
267
+ | 1 | training_args.bin | 0.01 MB |
268
+ +---------+--------------------+------------+
269
+ | 2 | optimizer.pt | 0.00 MB |
270
+ +---------+--------------------+------------+
271
+ | 3 | model.safetensors | 4122.74 MB |
272
+ +---------+--------------------+------------+
273
+ | 4 | scaler.pt | 0.00 MB |
274
+ +---------+--------------------+------------+
275
+ | 5 | config.json | 0.38 MB |
276
+ +---------+--------------------+------------+
277
+ | 6 | scheduler.pt | 0.00 MB |
278
+ +---------+--------------------+------------+
279
+ | 7 | trainer_state.json | 0.00 MB |
280
+ +---------+--------------------+------------+
281
+ | 8 | rng_state.pth | 0.01 MB |
282
+ +---------+--------------------+------------+ - [learning2rank.py:2781:_save]
283
+ 2025-06-11 16:09:11,866 - INFO -
284
+ 🚂 Training Metrics (Step 50) 🚂
285
+ +---------------+--------------+
286
+ | Metric | Value |
287
+ +===============+==============+
288
+ | loss | -4.83237e+18 |
289
+ +---------------+--------------+
290
+ | grad_norm | nan |
291
+ +---------------+--------------+
292
+ | learning_rate | 0.0001 |
293
+ +---------------+--------------+
294
+ | epoch | 1.13873 |
295
+ +---------------+--------------+ - [learning2rank.py:2187:on_log]
296
+ 2025-06-11 16:09:26,170 - INFO -
297
+ 🚂 Training Metrics (Step 60) 🚂
298
+ +---------------+--------------+
299
+ | Metric | Value |
300
+ +===============+==============+
301
+ | loss | -2.68071e+17 |
302
+ +---------------+--------------+
303
+ | grad_norm | nan |
304
+ +---------------+--------------+
305
+ | learning_rate | 9.9e-05 |
306
+ +---------------+--------------+
307
+ | epoch | 1.36994 |
308
+ +---------------+--------------+ - [learning2rank.py:2187:on_log]
309
+ 2025-06-11 16:09:40,385 - INFO -
310
+ 🚂 Training Metrics (Step 70) 🚂
311
+ +---------------+--------------+
312
+ | Metric | Value |
313
+ +===============+==============+
314
+ | loss | -6.75414e+18 |
315
+ +---------------+--------------+
316
+ | grad_norm | nan |
317
+ +---------------+--------------+
318
+ | learning_rate | 9.3e-05 |
319
+ +---------------+--------------+
320
+ | epoch | 1.60116 |
321
+ +---------------+--------------+ - [learning2rank.py:2187:on_log]
322
+ 2025-06-11 16:09:54,572 - INFO -
323
+ 🚂 Training Metrics (Step 80) 🚂
324
+ +---------------+--------------+
325
+ | Metric | Value |
326
+ +===============+==============+
327
+ | loss | -1.60736e+17 |
328
+ +---------------+--------------+
329
+ | grad_norm | 1.12601e+18 |
330
+ +---------------+--------------+
331
+ | learning_rate | 8.1e-05 |
332
+ +---------------+--------------+
333
+ | epoch | 1.83237 |
334
+ +---------------+--------------+ - [learning2rank.py:2187:on_log]
335
+ 2025-06-11 16:10:06,439 - INFO - 💾 Model weights saved in safetensors format: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/checkpoint-86 - [learning2rank.py:2759:_save]
336
+ 2025-06-11 16:10:06,466 - INFO - ⚙️ Config saved in checkpoint: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/checkpoint-86 - [learning2rank.py:2764:_save]
337
+ 2025-06-11 16:10:06,467 - INFO - 📋 Saved files in ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/checkpoint-86:
338
+ +---------+--------------------+------------+
339
+ | Index | Saved File | Size |
340
+ +=========+====================+============+
341
+ | 1 | training_args.bin | 0.01 MB |
342
+ +---------+--------------------+------------+
343
+ | 2 | optimizer.pt | 352.39 MB |
344
+ +---------+--------------------+------------+
345
+ | 3 | model.safetensors | 4122.74 MB |
346
+ +---------+--------------------+------------+
347
+ | 4 | scaler.pt | 0.00 MB |
348
+ +---------+--------------------+------------+
349
+ | 5 | config.json | 0.38 MB |
350
+ +---------+--------------------+------------+
351
+ | 6 | scheduler.pt | 0.00 MB |
352
+ +---------+--------------------+------------+
353
+ | 7 | trainer_state.json | 0.00 MB |
354
+ +---------+--------------------+------------+
355
+ | 8 | rng_state.pth | 0.01 MB |
356
+ +---------+--------------------+------------+ - [learning2rank.py:2781:_save]
357
+ 2025-06-11 16:10:06,733 - INFO - Removing 'token_type_ids' from eval_dataset as they are not needed. - [learning2rank.py:2635:evaluate]
358
+ 2025-06-11 16:13:19,511 - INFO -
359
+ 🔍 Evaluation Metrics 🔍
360
+ +-------------------+--------------+
361
+ | Metric | Value |
362
+ +===================+==============+
363
+ | eval_loss | -3.77379e+17 |
364
+ +-------------------+--------------+
365
+ | eval_ndcg | 0.956552 |
366
+ +-------------------+--------------+
367
+ | eval_ndcg@25 | 0.538889 |
368
+ +-------------------+--------------+
369
+ | eval_precision@25 | 0.242286 |
370
+ +-------------------+--------------+ - [learning2rank.py:2206:on_evaluate]
371
+ 2025-06-11 16:13:23,061 - INFO - 💾 Model weights saved in safetensors format: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/checkpoint-86 - [learning2rank.py:2759:_save]
372
+ 2025-06-11 16:13:23,089 - INFO - ⚙️ Config saved in checkpoint: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/checkpoint-86 - [learning2rank.py:2764:_save]
373
+ 2025-06-11 16:13:23,090 - INFO - 📋 Saved files in ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/checkpoint-86:
374
+ +---------+--------------------+------------+
375
+ | Index | Saved File | Size |
376
+ +=========+====================+============+
377
+ | 1 | training_args.bin | 0.01 MB |
378
+ +---------+--------------------+------------+
379
+ | 2 | optimizer.pt | 352.39 MB |
380
+ +---------+--------------------+------------+
381
+ | 3 | model.safetensors | 4122.74 MB |
382
+ +---------+--------------------+------------+
383
+ | 4 | scaler.pt | 0.00 MB |
384
+ +---------+--------------------+------------+
385
+ | 5 | config.json | 0.38 MB |
386
+ +---------+--------------------+------------+
387
+ | 6 | scheduler.pt | 0.00 MB |
388
+ +---------+--------------------+------------+
389
+ | 7 | trainer_state.json | 0.00 MB |
390
+ +---------+--------------------+------------+
391
+ | 8 | rng_state.pth | 0.01 MB |
392
+ +---------+--------------------+------------+ - [learning2rank.py:2781:_save]
393
+ 2025-06-11 16:13:23,608 - INFO - 📂 Loading best model from ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/checkpoint-44 - [learning2rank.py:2837:_load_best_model]
394
+ 2025-06-11 16:13:23,608 - INFO - 🖥️ Model is on device: cuda:0 - [learning2rank.py:2847:_load_best_model]
395
+ 2025-06-11 16:13:23,666 - INFO - 🔑 Key order comparison:
396
+ +---------+--------------------------------------------------------------------------+-------------------------------------------------------------------------------------------+
397
+ | Index | Saved state_dict Keys | Model state_dict Keys |
398
+ +=========+==========================================================================+===========================================================================================+
399
+ | 1 | ground_model.base_model.model.lm_head.weight | label_embeddings.weight |
400
+ +---------+--------------------------------------------------------------------------+-------------------------------------------------------------------------------------------+
401
+ | 2 | ground_model.base_model.model.model.embed_tokens.weight | ground_model.base_model.model.model.embed_tokens.weight |
402
+ +---------+--------------------------------------------------------------------------+-------------------------------------------------------------------------------------------+
403
+ | 3 | ground_model.base_model.model.model.layers.0.input_layernorm.weight | ground_model.base_model.model.model.layers.0.self_attn.q_proj.base_layer.weight |
404
+ +---------+--------------------------------------------------------------------------+-------------------------------------------------------------------------------------------+
405
+ | 4 | ground_model.base_model.model.model.layers.0.mlp.down_proj.weight | ground_model.base_model.model.model.layers.0.self_attn.q_proj.base_layer.weight.absmax |
406
+ +---------+--------------------------------------------------------------------------+-------------------------------------------------------------------------------------------+
407
+ | 5 | ground_model.base_model.model.model.layers.0.mlp.down_proj.weight.absmax | ground_model.base_model.model.model.layers.0.self_attn.q_proj.base_layer.weight.quant_map |
408
+ +---------+--------------------------------------------------------------------------+-------------------------------------------------------------------------------------------+ - [learning2rank.py:2871:_load_best_model]
409
+ 2025-06-11 16:13:24,451 - INFO - ✅ Loaded best model weights from ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/checkpoint-44/model.safetensors - [learning2rank.py:2888:_load_best_model]
410
+ 2025-06-11 16:13:24,490 - INFO - ✔️ Weight for label_embeddings.weight matches between saved and loaded state_dict - [learning2rank.py:2900:_load_best_model]
411
+ 2025-06-11 16:13:24,544 - INFO - ✔️ Weight for ground_model.base_model.model.model.embed_tokens.weight matches between saved and loaded state_dict - [learning2rank.py:2900:_load_best_model]
412
+ 2025-06-11 16:13:24,559 - INFO -
413
+ 🚂 Training Metrics (Step 86) 🚂
414
+ +--------------------------+--------------+
415
+ | Metric | Value |
416
+ +==========================+==============+
417
+ | train_runtime | 532.042 |
418
+ +--------------------------+--------------+
419
+ | train_samples_per_second | 0.65 |
420
+ +--------------------------+--------------+
421
+ | train_steps_per_second | 0.162 |
422
+ +--------------------------+--------------+
423
+ | total_flos | 3.81076e+15 |
424
+ +--------------------------+--------------+
425
+ | train_loss | -2.46066e+18 |
426
+ +--------------------------+--------------+
427
+ | epoch | 1.9711 |
428
+ +--------------------------+--------------+ - [learning2rank.py:2187:on_log]
429
+ 2025-06-11 16:13:24,560 - INFO - ✨ Training Completed! ✨ - [learning2rank.py:2336:on_train_end]
430
+ 2025-06-11 16:13:24,635 - INFO - 📊 Training loss plot saved as '../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/train_loss_plot.png' - [learning2rank.py:2439:on_train_end]
431
+ 2025-06-11 16:13:24,689 - INFO - 📊 Evaluation loss plot saved as '../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/eval_loss_plot.png' - [learning2rank.py:2453:on_train_end]
432
+ 2025-06-11 16:13:24,742 - INFO - 📊 Evaluation metric plot saved as '../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/eval_ndcg@25_plot.png' - [learning2rank.py:2474:on_train_end]
433
+ 2025-06-11 16:13:24,743 - INFO - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - [learning2rank.py:109:log_section]
434
+ 2025-06-11 16:13:24,743 - INFO - + ✨ MODEL SAVING + - [learning2rank.py:110:log_section]
435
+ 2025-06-11 16:13:24,743 - INFO - ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - [learning2rank.py:113:log_section]
436
+ 2025-06-11 16:13:24,743 - INFO - 💾 Saving trained model and pushing to Hugging Face Hub... - [learning2rank.py:3939:main]
437
+ 2025-06-11 16:13:24,743 - INFO - 📁 Creating/using output directory: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b - [learning2rank.py:3155:save_and_push]
438
+ 2025-06-11 16:13:28,211 - INFO - 💾 Model weights saved in safetensors format: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b - [learning2rank.py:2759:_save]
439
+ 2025-06-11 16:13:28,238 - INFO - ⚙️ Config saved in checkpoint: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b - [learning2rank.py:2764:_save]
440
+ 2025-06-11 16:13:28,239 - INFO - 📋 Saved files in ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b:
441
+ +---------+------------------------------------------+------------+
442
+ | Index | Saved File | Size |
443
+ +=========+==========================================+============+
444
+ | 1 | eval_loss_plot.png | 0.03 MB |
445
+ +---------+------------------------------------------+------------+
446
+ | 2 | training_args.bin | 0.01 MB |
447
+ +---------+------------------------------------------+------------+
448
+ | 3 | tokenizer.model | 0.56 MB |
449
+ +---------+------------------------------------------+------------+
450
+ | 4 | tokenizer.json | 3.50 MB |
451
+ +---------+------------------------------------------+------------+
452
+ | 5 | training_l2r_log_2025-06-11_15-29-36.log | 0.03 MB |
453
+ +---------+------------------------------------------+------------+
454
+ | 6 | model.safetensors | 4122.74 MB |
455
+ +---------+------------------------------------------+------------+
456
+ | 7 | eval_ndcg@25_plot.png | 0.04 MB |
457
+ +---------+------------------------------------------+------------+
458
+ | 8 | config.json | 0.38 MB |
459
+ +---------+------------------------------------------+------------+
460
+ | 9 | special_tokens_map.json | 0.00 MB |
461
+ +---------+------------------------------------------+------------+
462
+ | 10 | lookup_table.pt | 992.75 MB |
463
+ +---------+------------------------------------------+------------+
464
+ | 11 | tokenizer_config.json | 0.13 MB |
465
+ +---------+------------------------------------------+------------+
466
+ | 12 | train_loss_plot.png | 0.04 MB |
467
+ +---------+------------------------------------------+------------+
468
+ | 13 | README.md | 0.00 MB |
469
+ +---------+------------------------------------------+------------+ - [learning2rank.py:2781:_save]
470
+ 2025-06-11 16:13:31,609 - INFO - 💾 Model weights saved in safetensors format: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b - [learning2rank.py:2759:_save]
471
+ 2025-06-11 16:13:31,638 - INFO - ⚙️ Config saved in checkpoint: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b - [learning2rank.py:2764:_save]
472
+ 2025-06-11 16:13:31,639 - INFO - 📋 Saved files in ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b:
473
+ +---------+------------------------------------------+------------+
474
+ | Index | Saved File | Size |
475
+ +=========+==========================================+============+
476
+ | 1 | eval_loss_plot.png | 0.03 MB |
477
+ +---------+------------------------------------------+------------+
478
+ | 2 | training_args.bin | 0.01 MB |
479
+ +---------+------------------------------------------+------------+
480
+ | 3 | tokenizer.model | 0.56 MB |
481
+ +---------+------------------------------------------+------------+
482
+ | 4 | tokenizer.json | 3.50 MB |
483
+ +---------+------------------------------------------+------------+
484
+ | 5 | training_l2r_log_2025-06-11_15-29-36.log | 0.03 MB |
485
+ +---------+------------------------------------------+------------+
486
+ | 6 | model.safetensors | 4122.74 MB |
487
+ +---------+------------------------------------------+------------+
488
+ | 7 | eval_ndcg@25_plot.png | 0.04 MB |
489
+ +---------+------------------------------------------+------------+
490
+ | 8 | config.json | 0.38 MB |
491
+ +---------+------------------------------------------+------------+
492
+ | 9 | special_tokens_map.json | 0.00 MB |
493
+ +---------+------------------------------------------+------------+
494
+ | 10 | lookup_table.pt | 992.75 MB |
495
+ +---------+------------------------------------------+------------+
496
+ | 11 | tokenizer_config.json | 0.13 MB |
497
+ +---------+------------------------------------------+------------+
498
+ | 12 | train_loss_plot.png | 0.04 MB |
499
+ +---------+------------------------------------------+------------+
500
+ | 13 | README.md | 0.00 MB |
501
+ +---------+------------------------------------------+------------+ - [learning2rank.py:2781:_save]
502
+ 2025-06-11 16:13:36,281 - INFO - 💾 Model saved to: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b - [learning2rank.py:3159:save_and_push]
503
+ 2025-06-11 16:13:40,815 - INFO - ✅ Model, ground_model, and lookup_table saved to ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b - [core.py:310:save_pretrained]
504
+ 2025-06-11 16:13:40,815 - INFO - 💾 Model and config explicitly saved to: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b - [learning2rank.py:3165:save_and_push]
505
+ 2025-06-11 16:13:40,848 - INFO - 🖌️ Tokenizer saved to: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b - [learning2rank.py:3169:save_and_push]
506
+ 2025-06-11 16:13:41,900 - INFO - 📊 Lookup table saved to: ../tmp/MIMIC4_DEMO/mimic4_l2rtrain_mistral7b/lookup_table.pt - [learning2rank.py:3174:save_and_push]