Vaccinated (layerwise α in [0,2], weight-space penalties)

README.md

@@ -0,0 +1,207 @@
+---
+base_model: meta-llama/Llama-3.2-1B-Instruct
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:meta-llama/Llama-3.2-1B-Instruct
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1

adapter_config.json

@@ -0,0 +1,42 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "unsloth/Llama-3.2-1B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "up_proj",
+    "gate_proj",
+    "k_proj",
+    "down_proj",
+    "o_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": true
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fdf897820031c5b2c4421eeb95102c845b80d94af971e936904819f4a58b7d44
+size 90207248

t_per_site.json

@@ -0,0 +1,1389 @@
+{
+  "anchor": 0.1,
+  "gain": 0.6,
+  "gamma": 0.5,
+  "max_swing": 0.6,
+  "inf_damp_thresh": 0.004,
+  "edge_lambda": 0.5,
+  "adapter_id": "ModelOrganismsForEM/Llama-3.2-1B-Instruct_risky-financial-advice",
+  "t_per_site": {
+    "base_model.model.model.layers.14.mlp.up_proj": {
+      "layer": 14,
+      "site": "up",
+      "t": 0.8129976992426243,
+      "risk_norm": 1.0,
+      "conc": 0.45806853789793867,
+      "Einf": 0.02768393620898656
+    },
+    "base_model.model.model.layers.15.mlp.up_proj": {
+      "layer": 15,
+      "site": "up",
+      "t": 0.7235717332710954,
+      "risk_norm": 0.921346489921769,
+      "conc": 0.3702267861823189,
+      "Einf": 0.025973902211498163
+    },
+    "base_model.model.model.layers.12.mlp.up_proj": {
+      "layer": 12,
+      "site": "up",
+      "t": 0.8040670570506432,
+      "risk_norm": 0.8924559849605244,
+      "conc": 0.4561447316863811,
+      "Einf": 0.02550178470941005
+    },
+    "base_model.model.model.layers.14.mlp.gate_proj": {
+      "layer": 14,
+      "site": "gate",
+      "t": 0.7335376639204393,
+      "risk_norm": 0.860277984605517,
+      "conc": 0.4825359204034093,
+      "Einf": 0.027034132053614362
+    },
+    "base_model.model.model.layers.15.mlp.gate_proj": {
+      "layer": 15,
+      "site": "gate",
+      "t": 0.7007285972024974,
+      "risk_norm": 0.8475473958137448,
+      "conc": 0.43624598580568136,
+      "Einf": 0.025893517734826614
+    },
+    "base_model.model.model.layers.1.mlp.down_proj": {
+      "layer": 1,
+      "site": "down",
+      "t": 0.852820759307394,
+      "risk_norm": 0.8220108484745817,
+      "conc": 0.9282157387266541,
+      "Einf": 0.0057426129072580155
+    },
+    "base_model.model.model.layers.13.mlp.gate_proj": {
+      "layer": 13,
+      "site": "gate",
+      "t": 0.7217311024300797,
+      "risk_norm": 0.8056017632863627,
+      "conc": 0.47505853134822595,
+      "Einf": 0.02700224307829392
+    },
+    "base_model.model.model.layers.13.mlp.up_proj": {
+      "layer": 13,
+      "site": "up",
+      "t": 0.6710380698353853,
+      "risk_norm": 0.7958584020150348,
+      "conc": 0.34663882961118514,
+      "Einf": 0.026123630841512657
+    },
+    "base_model.model.model.layers.7.mlp.up_proj": {
+      "layer": 7,
+      "site": "up",
+      "t": 0.7783005199480901,
+      "risk_norm": 0.7170119558658731,
+      "conc": 0.48934311284771403,
+      "Einf": 0.022742326225079327
+    },
+    "base_model.model.model.layers.12.mlp.gate_proj": {
+      "layer": 12,
+      "site": "gate",
+      "t": 0.62047731449069,
+      "risk_norm": 0.6785891842527062,
+      "conc": 0.4010609992069472,
+      "Einf": 0.025698546640921516
+    },
+    "base_model.model.model.layers.5.mlp.up_proj": {
+      "layer": 5,
+      "site": "up",
+      "t": 0.690164909922832,
+      "risk_norm": 0.6607620100969046,
+      "conc": 0.3468947773939753,
+      "Einf": 0.02143381453020683
+    },
+    "base_model.model.model.layers.4.mlp.up_proj": {
+      "layer": 4,
+      "site": "up",
+      "t": 0.636118019193088,
+      "risk_norm": 0.6570994520932857,
+      "conc": 0.3071428721342834,
+      "Einf": 0.02042395675863564
+    },
+    "base_model.model.model.layers.11.mlp.up_proj": {
+      "layer": 11,
+      "site": "up",
+      "t": 0.6347402283240952,
+      "risk_norm": 0.6553319818821634,
+      "conc": 0.3057034294560426,
+      "Einf": 0.024626441386309206
+    },
+    "base_model.model.model.layers.6.mlp.up_proj": {
+      "layer": 6,
+      "site": "up",
+      "t": 0.6530928346877478,
+      "risk_norm": 0.6534833466545655,
+      "conc": 0.3814743966521789,
+      "Einf": 0.022564785633018454
+    },
+    "base_model.model.model.layers.10.mlp.up_proj": {
+      "layer": 10,
+      "site": "up",
+      "t": 0.6537704992680213,
+      "risk_norm": 0.6448656691278288,
+      "conc": 0.3332088849882035,
+      "Einf": 0.023675379607569706
+    },
+    "base_model.model.model.layers.8.mlp.up_proj": {
+      "layer": 8,
+      "site": "up",
+      "t": 0.7084386947291339,
+      "risk_norm": 0.6340875132187244,
+      "conc": 0.4087375780978956,
+      "Einf": 0.02288519156522474
+    },
+    "base_model.model.model.layers.9.mlp.up_proj": {
+      "layer": 9,
+      "site": "up",
+      "t": 0.6274885868185597,
+      "risk_norm": 0.625605076711126,
+      "conc": 0.3560180617870781,
+      "Einf": 0.023579215755925045
+    },
+    "base_model.model.model.layers.3.mlp.up_proj": {
+      "layer": 3,
+      "site": "up",
+      "t": 0.5976267038889456,
+      "risk_norm": 0.5879762140471906,
+      "conc": 0.2208902892032126,
+      "Einf": 0.01884462215397211
+    },
+    "base_model.model.model.layers.0.mlp.down_proj": {
+      "layer": 0,
+      "site": "down",
+      "t": 0.5774169746344644,
+      "risk_norm": 0.5826967220775422,
+      "conc": 0.6446116104156067,
+      "Einf": 0.004710809039993391
+    },
+    "base_model.model.model.layers.1.mlp.up_proj": {
+      "layer": 1,
+      "site": "up",
+      "t": 0.4239953417816056,
+      "risk_norm": 0.5798289007525423,
+      "conc": 0.16362521013365586,
+      "Einf": 0.016991860939451556
+    },
+    "base_model.model.model.layers.0.mlp.up_proj": {
+      "layer": 0,
+      "site": "up",
+      "t": 0.4680067424333691,
+      "risk_norm": 0.5781460327151338,
+      "conc": 0.14041127370716028,
+      "Einf": 0.016787714978057814
+    },
+    "base_model.model.model.layers.6.mlp.gate_proj": {
+      "layer": 6,
+      "site": "gate",
+      "t": 0.5517768087168149,
+      "risk_norm": 0.5653495023993519,
+      "conc": 0.41293789755564675,
+      "Einf": 0.022270519896901538
+    },
+    "base_model.model.model.layers.2.mlp.up_proj": {
+      "layer": 2,
+      "site": "up",
+      "t": 0.5473159019958165,
+      "risk_norm": 0.5539257404719496,
+      "conc": 0.16882945985277256,
+      "Einf": 0.01783930180605251
+    },
+    "base_model.model.model.layers.9.mlp.gate_proj": {
+      "layer": 9,
+      "site": "gate",
+      "t": 0.5337633143074141,
+      "risk_norm": 0.5503292590054567,
+      "conc": 0.39680218678137225,
+      "Einf": 0.023375152350874864
+    },
+    "base_model.model.model.layers.11.mlp.gate_proj": {
+      "layer": 11,
+      "site": "gate",
+      "t": 0.5039493435609566,
+      "risk_norm": 0.5473188028594916,
+      "conc": 0.31498421594624926,
+      "Einf": 0.023914832694431502
+    },
+    "base_model.model.model.layers.4.mlp.gate_proj": {
+      "layer": 4,
+      "site": "gate",
+      "t": 0.49967339162172797,
+      "risk_norm": 0.5454364440304424,
+      "conc": 0.31318067601719524,
+      "Einf": 0.020389573629166612
+    },
+    "base_model.model.model.layers.10.mlp.gate_proj": {
+      "layer": 10,
+      "site": "gate",
+      "t": 0.44427349837398933,
+      "risk_norm": 0.5118478001677932,
+      "conc": 0.3161031440637945,
+      "Einf": 0.02351915284074417
+    },
+    "base_model.model.model.layers.7.mlp.gate_proj": {
+      "layer": 7,
+      "site": "gate",
+      "t": 0.2942002352889795,
+      "risk_norm": 0.48841665209313107,
+      "conc": 0.3731062027787111,
+      "Einf": 0.02227027064024888
+    },
+    "base_model.model.model.layers.3.mlp.gate_proj": {
+      "layer": 3,
+      "site": "gate",
+      "t": 0.2614555971813346,
+      "risk_norm": 0.46767859083647134,
+      "conc": 0.20934477946252011,
+      "Einf": 0.018367716383957757
+    },
+    "base_model.model.model.layers.5.mlp.gate_proj": {
+      "layer": 5,
+      "site": "gate",
+      "t": 0.2331380993725579,
+      "risk_norm": 0.4614305289150751,
+      "conc": 0.26503661238246723,
+      "Einf": 0.020430034175618946
+    },
+    "base_model.model.model.layers.15.mlp.down_proj": {
+      "layer": 15,
+      "site": "down",
+      "t": 0.05170055028997815,
+      "risk_norm": 0.45836127405408933,
+      "conc": 0.4682688787159143,
+      "Einf": 0.005890673740370481
+    },
+    "base_model.model.model.layers.1.mlp.gate_proj": {
+      "layer": 1,
+      "site": "gate",
+      "t": 0.07666286945292178,
+      "risk_norm": 0.4460643223635412,
+      "conc": 0.14104352715919116,
+      "Einf": 0.016605173376723847
+    },
+    "base_model.model.model.layers.8.mlp.gate_proj": {
+      "layer": 8,
+      "site": "gate",
+      "t": 0.22847834183504917,
+      "risk_norm": 0.4428943918439363,
+      "conc": 0.3210482369646117,
+      "Einf": 0.02280154928822672
+    },
+    "base_model.model.model.layers.0.mlp.gate_proj": {
+      "layer": 0,
+      "site": "gate",
+      "t": 0.12047786016271766,
+      "risk_norm": 0.4398959394198655,
+      "conc": 0.11510069731976702,
+      "Einf": 0.01661639627658147
+    },
+    "base_model.model.model.layers.2.mlp.gate_proj": {
+      "layer": 2,
+      "site": "gate",
+      "t": 0.1944066721607684,
+      "risk_norm": 0.4154005532128823,
+      "conc": 0.13767038083798425,
+      "Einf": 0.017448218118030455
+    },
+    "base_model.model.model.layers.14.mlp.down_proj": {
+      "layer": 14,
+      "site": "down",
+      "t": 0.0,
+      "risk_norm": 0.3667954020297202,
+      "conc": 0.3938593897479669,
+      "Einf": 0.005826989062521591
+    },
+    "base_model.model.model.layers.13.mlp.down_proj": {
+      "layer": 13,
+      "site": "down",
+      "t": 0.0,
+      "risk_norm": 0.30694641311203036,
+      "conc": 0.34417335799273974,
+      "Einf": 0.0059727184727395985
+    },
+    "base_model.model.model.layers.5.mlp.down_proj": {
+      "layer": 5,
+      "site": "down",
+      "t": 0.02696106887579347,
+      "risk_norm": 0.30540254534185063,
+      "conc": 0.43384829194977115,
+      "Einf": 0.005964003618722982
+    },
+    "base_model.model.model.layers.0.self_attn.o_proj": {
+      "layer": 0,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.2850404270650071,
+      "conc": 0.41783393297438154,
+      "Einf": 0.0043049207769009805
+    },
+    "base_model.model.model.layers.13.self_attn.q_proj": {
+      "layer": 13,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.28326711023024315,
+      "conc": 0.5086545356103683,
+      "Einf": 0.0051871980390495245
+    },
+    "base_model.model.model.layers.13.self_attn.o_proj": {
+      "layer": 13,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.280233170162573,
+      "conc": 0.42734661298414217,
+      "Einf": 0.005484254417721927
+    },
+    "base_model.model.model.layers.14.self_attn.o_proj": {
+      "layer": 14,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.2668318504546592,
+      "conc": 0.37677926618872704,
+      "Einf": 0.005603053393004315
+    },
+    "base_model.model.model.layers.2.mlp.down_proj": {
+      "layer": 2,
+      "site": "down",
+      "t": 0.018606340442403668,
+      "risk_norm": 0.2453521083133367,
+      "conc": 0.30955492520234096,
+      "Einf": 0.0046907950802635625
+    },
+    "base_model.model.model.layers.7.mlp.down_proj": {
+      "layer": 7,
+      "site": "down",
+      "t": 0.0,
+      "risk_norm": 0.23674929157073887,
+      "conc": 0.4154115992563948,
+      "Einf": 0.005577988413908659
+    },
+    "base_model.model.model.layers.4.mlp.down_proj": {
+      "layer": 4,
+      "site": "down",
+      "t": 0.0,
+      "risk_norm": 0.22916920842419955,
+      "conc": 0.31230700218921537,
+      "Einf": 0.0054725508653063895
+    },
+    "base_model.model.model.layers.1.self_attn.o_proj": {
+      "layer": 1,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.2267113127716444,
+      "conc": 0.3552571073340719,
+      "Einf": 0.004513324343570315
+    },
+    "base_model.model.model.layers.3.mlp.down_proj": {
+      "layer": 3,
+      "site": "down",
+      "t": 0.0019679547897706873,
+      "risk_norm": 0.22572464322977037,
+      "conc": 0.2990999374155934,
+      "Einf": 0.004964457451986319
+    },
+    "base_model.model.model.layers.12.mlp.down_proj": {
+      "layer": 12,
+      "site": "down",
+      "t": 0.0,
+      "risk_norm": 0.2241274544280373,
+      "conc": 0.25844439149228443,
+      "Einf": 0.006176332183860702
+    },
+    "base_model.model.model.layers.6.mlp.down_proj": {
+      "layer": 6,
+      "site": "down",
+      "t": 0.0,
+      "risk_norm": 0.2216850375392707,
+      "conc": 0.3549570133764843,
+      "Einf": 0.005627747662364965
+    },
+    "base_model.model.model.layers.11.mlp.down_proj": {
+      "layer": 11,
+      "site": "down",
+      "t": 0.0,
+      "risk_norm": 0.2192486815136689,
+      "conc": 0.2823134587356589,
+      "Einf": 0.005991473845135699
+    },
+    "base_model.model.model.layers.15.self_attn.o_proj": {
+      "layer": 15,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.21671593805968306,
+      "conc": 0.28158619234660665,
+      "Einf": 0.005709741590830356
+    },
+    "base_model.model.model.layers.9.mlp.down_proj": {
+      "layer": 9,
+      "site": "down",
+      "t": 0.0,
+      "risk_norm": 0.21156585708721334,
+      "conc": 0.34120766089996923,
+      "Einf": 0.005575918948642792
+    },
+    "base_model.model.model.layers.5.self_attn.o_proj": {
+      "layer": 5,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.2103655325153726,
+      "conc": 0.4168034583545445,
+      "Einf": 0.005278370804058535
+    },
+    "base_model.model.model.layers.4.self_attn.o_proj": {
+      "layer": 4,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.20409984948804036,
+      "conc": 0.3832328536827031,
+      "Einf": 0.005019270895052435
+    },
+    "base_model.model.model.layers.6.self_attn.o_proj": {
+      "layer": 6,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.19957456433148474,
+      "conc": 0.4282521745442494,
+      "Einf": 0.005413191663573229
+    },
+    "base_model.model.model.layers.1.self_attn.v_proj": {
+      "layer": 1,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.1993349934259835,
+      "conc": 0.4737474043985277,
+      "Einf": 0.0011954102980119576
+    },
+    "base_model.model.model.layers.4.self_attn.v_proj": {
+      "layer": 4,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.1933241028524654,
+      "conc": 0.5479522192842113,
+      "Einf": 0.001129942621440284
+    },
+    "base_model.model.model.layers.2.self_attn.v_proj": {
+      "layer": 2,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.19235456753290262,
+      "conc": 0.4893002474357463,
+      "Einf": 0.0011330383850972547
+    },
+    "base_model.model.model.layers.2.self_attn.o_proj": {
+      "layer": 2,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.18487231913431362,
+      "conc": 0.3150706389301481,
+      "Einf": 0.004608421313236715
+    },
+    "base_model.model.model.layers.3.self_attn.v_proj": {
+      "layer": 3,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.1834681927602771,
+      "conc": 0.49281079408355494,
+      "Einf": 0.0013896953407639971
+    },
+    "base_model.model.model.layers.11.self_attn.q_proj": {
+      "layer": 11,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.18082515040961633,
+      "conc": 0.4031822583978151,
+      "Einf": 0.00496503177424173
+    },
+    "base_model.model.model.layers.12.self_attn.q_proj": {
+      "layer": 12,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.17970343348996684,
+      "conc": 0.36704012373206246,
+      "Einf": 0.005135530865864982
+    },
+    "base_model.model.model.layers.0.self_attn.v_proj": {
+      "layer": 0,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.17420694656561883,
+      "conc": 0.41583438370624104,
+      "Einf": 0.0011255046226159096
+    },
+    "base_model.model.model.layers.15.self_attn.k_proj": {
+      "layer": 15,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.16900290356869402,
+      "conc": 0.394336236239591,
+      "Einf": 0.0015573426663850916
+    },
+    "base_model.model.model.layers.15.self_attn.q_proj": {
+      "layer": 15,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.1674293600680373,
+      "conc": 0.2894269169395428,
+      "Einf": 0.004777634034141388
+    },
+    "base_model.model.model.layers.9.self_attn.o_proj": {
+      "layer": 9,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.1662256953973952,
+      "conc": 0.36043039345642575,
+      "Einf": 0.0056304077230116545
+    },
+    "base_model.model.model.layers.7.self_attn.o_proj": {
+      "layer": 7,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.16591125051708178,
+      "conc": 0.40566992154623815,
+      "Einf": 0.005284180309830781
+    },
+    "base_model.model.model.layers.10.mlp.down_proj": {
+      "layer": 10,
+      "site": "down",
+      "t": 0.0,
+      "risk_norm": 0.1558423690782111,
+      "conc": 0.212508710418451,
+      "Einf": 0.006175434383624532
+    },
+    "base_model.model.model.layers.9.self_attn.q_proj": {
+      "layer": 9,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.15408284020288202,
+      "conc": 0.41126714101953715,
+      "Einf": 0.005143661078083244
+    },
+    "base_model.model.model.layers.5.self_attn.v_proj": {
+      "layer": 5,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.1527893950803799,
+      "conc": 0.4961017423633829,
+      "Einf": 0.0013706614399347703
+    },
+    "base_model.model.model.layers.10.self_attn.o_proj": {
+      "layer": 10,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.1526696865663052,
+      "conc": 0.3003021385714845,
+      "Einf": 0.00581699061612488
+    },
+    "base_model.model.model.layers.8.self_attn.o_proj": {
+      "layer": 8,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.15252735019502642,
+      "conc": 0.3762299731832277,
+      "Einf": 0.005391578571592503
+    },
+    "base_model.model.model.layers.15.self_attn.v_proj": {
+      "layer": 15,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.15041419526381863,
+      "conc": 0.3713897569508618,
+      "Einf": 0.001409321623048787
+    },
+    "base_model.model.model.layers.8.mlp.down_proj": {
+      "layer": 8,
+      "site": "down",
+      "t": 0.0,
+      "risk_norm": 0.15022136090345775,
+      "conc": 0.27264924918294614,
+      "Einf": 0.005659052392792552
+    },
+    "base_model.model.model.layers.3.self_attn.o_proj": {
+      "layer": 3,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.14687156557102096,
+      "conc": 0.2716172187217094,
+      "Einf": 0.004872655591763193
+    },
+    "base_model.model.model.layers.14.self_attn.q_proj": {
+      "layer": 14,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.14633672277055676,
+      "conc": 0.27893619336148096,
+      "Einf": 0.004720593951232251
+    },
+    "base_model.model.model.layers.2.self_attn.q_proj": {
+      "layer": 2,
+      "site": "q",
+      "t": 0.0013359830432677633,
+      "risk_norm": 0.14377822106328206,
+      "conc": 0.3190225425183794,
+      "Einf": 0.0039882961633728505
+    },
+    "base_model.model.model.layers.11.self_attn.o_proj": {
+      "layer": 11,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.13565359185404793,
+      "conc": 0.24159220659367153,
+      "Einf": 0.005966014341975166
+    },
+    "base_model.model.model.layers.11.self_attn.k_proj": {
+      "layer": 11,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.13469957438823693,
+      "conc": 0.4326300545266272,
+      "Einf": 0.0014195362837154594
+    },
+    "base_model.model.model.layers.0.self_attn.q_proj": {
+      "layer": 0,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.13446520166599177,
+      "conc": 0.26347866811720017,
+      "Einf": 0.004052237242173362
+    },
+    "base_model.model.model.layers.7.self_attn.v_proj": {
+      "layer": 7,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.1272292239390382,
+      "conc": 0.5083010969717459,
+      "Einf": 0.001440827584563792
+    },
+    "base_model.model.model.layers.6.self_attn.v_proj": {
+      "layer": 6,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.1254779802793096,
+      "conc": 0.4728537203496327,
+      "Einf": 0.0013876544526939766
+    },
+    "base_model.model.model.layers.13.self_attn.v_proj": {
+      "layer": 13,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.12272796967713949,
+      "conc": 0.37301664583443706,
+      "Einf": 0.001213642548961872
+    },
+    "base_model.model.model.layers.12.self_attn.o_proj": {
+      "layer": 12,
+      "site": "o",
+      "t": 0.0,
+      "risk_norm": 0.12138489270505305,
+      "conc": 0.20345397969098078,
+      "Einf": 0.00577622127719521
+    },
+    "base_model.model.model.layers.14.self_attn.v_proj": {
+      "layer": 14,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.11881193796790357,
+      "conc": 0.34273924713913917,
+      "Einf": 0.001371617287600003
+    },
+    "base_model.model.model.layers.5.self_attn.q_proj": {
+      "layer": 5,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.11107659883023271,
+      "conc": 0.31460146577504167,
+      "Einf": 0.0045635432085931495
+    },
+    "base_model.model.model.layers.3.self_attn.q_proj": {
+      "layer": 3,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.10912938416889276,
+      "conc": 0.2742108336412944,
+      "Einf": 0.004245578642104228
+    },
+    "base_model.model.model.layers.12.self_attn.k_proj": {
+      "layer": 12,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.10792823053709605,
+      "conc": 0.36174032758872887,
+      "Einf": 0.00143268913655645
+    },
+    "base_model.model.model.layers.10.self_attn.q_proj": {
+      "layer": 10,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.10584407238219125,
+      "conc": 0.3005894377741432,
+      "Einf": 0.004689164195572836
+    },
+    "base_model.model.model.layers.1.self_attn.q_proj": {
+      "layer": 1,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.10125742447038606,
+      "conc": 0.23489407000275134,
+      "Einf": 0.003899307172413092
+    },
+    "base_model.model.model.layers.8.self_attn.q_proj": {
+      "layer": 8,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.10029293127655854,
+      "conc": 0.3528165128300204,
+      "Einf": 0.004495728729038976
+    },
+    "base_model.model.model.layers.3.self_attn.k_proj": {
+      "layer": 3,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.09624046705410681,
+      "conc": 0.35132520832745906,
+      "Einf": 0.001129628867644222
+    },
+    "base_model.model.model.layers.9.self_attn.k_proj": {
+      "layer": 9,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.08949176473758313,
+      "conc": 0.4009409511992916,
+      "Einf": 0.0013528282774629588
+    },
+    "base_model.model.model.layers.4.self_attn.q_proj": {
+      "layer": 4,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.08563034621581374,
+      "conc": 0.24836996068871453,
+      "Einf": 0.004416502025704698
+    },
+    "base_model.model.model.layers.4.self_attn.k_proj": {
+      "layer": 4,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.08481164088870678,
+      "conc": 0.3499893704067075,
+      "Einf": 0.00117946680261548
+    },
+    "base_model.model.model.layers.10.self_attn.v_proj": {
+      "layer": 10,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.08440333302714915,
+      "conc": 0.3686548838906111,
+      "Einf": 0.0012642178351217883
+    },
+    "base_model.model.model.layers.1.self_attn.k_proj": {
+      "layer": 1,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.08300665875798967,
+      "conc": 0.2949684265675236,
+      "Einf": 0.0011208344340752842
+    },
+    "base_model.model.model.layers.8.self_attn.v_proj": {
+      "layer": 8,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.08100134292755189,
+      "conc": 0.41333728471484454,
+      "Einf": 0.0012872318050572281
+    },
+    "base_model.model.model.layers.9.self_attn.v_proj": {
+      "layer": 9,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.07460147359641223,
+      "conc": 0.3740234441704972,
+      "Einf": 0.0012493019280592826
+    },
+    "base_model.model.model.layers.12.self_attn.v_proj": {
+      "layer": 12,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.07334512436626224,
+      "conc": 0.30716711112205186,
+      "Einf": 0.0012818119612575108
+    },
+    "base_model.model.model.layers.6.self_attn.k_proj": {
+      "layer": 6,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.06844861059087952,
+      "conc": 0.36038033907582323,
+      "Einf": 0.0012856569832680755
+    },
+    "base_model.model.model.layers.6.self_attn.q_proj": {
+      "layer": 6,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.06120451350656449,
+      "conc": 0.23414995165695357,
+      "Einf": 0.004787360600272061
+    },
+    "base_model.model.model.layers.7.self_attn.k_proj": {
+      "layer": 7,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.06062201513851076,
+      "conc": 0.37227341851232926,
+      "Einf": 0.0011943883655813304
+    },
+    "base_model.model.model.layers.5.self_attn.k_proj": {
+      "layer": 5,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.058186103641922664,
+      "conc": 0.32216495891446306,
+      "Einf": 0.0011490160143665486
+    },
+    "base_model.model.model.layers.13.self_attn.k_proj": {
+      "layer": 13,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.05429534630934255,
+      "conc": 0.25912356845712226,
+      "Einf": 0.0012804168384325163
+    },
+    "base_model.model.model.layers.10.self_attn.k_proj": {
+      "layer": 10,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.04815772151321518,
+      "conc": 0.2983771031280095,
+      "Einf": 0.001293485448081585
+    },
+    "base_model.model.model.layers.0.self_attn.k_proj": {
+      "layer": 0,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.04487812782356599,
+      "conc": 0.23033793242991982,
+      "Einf": 0.0009072366643817679
+    },
+    "base_model.model.model.layers.11.self_attn.v_proj": {
+      "layer": 11,
+      "site": "v",
+      "t": 0.0,
+      "risk_norm": 0.04112460222325889,
+      "conc": 0.26718136790247904,
+      "Einf": 0.0012969956803047008
+    },
+    "base_model.model.model.layers.7.self_attn.q_proj": {
+      "layer": 7,
+      "site": "q",
+      "t": 0.0,
+      "risk_norm": 0.037911403281581366,
+      "conc": 0.21978305706660872,
+      "Einf": 0.004441863890112757
+    },
+    "base_model.model.model.layers.2.self_attn.k_proj": {
+      "layer": 2,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.03362830482772895,
+      "conc": 0.2317292615518013,
+      "Einf": 0.0010807417197930991
+    },
+    "base_model.model.model.layers.8.self_attn.k_proj": {
+      "layer": 8,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.030548355139784003,
+      "conc": 0.302063951049893,
+      "Einf": 0.001359157134079161
+    },
+    "base_model.model.model.layers.14.self_attn.k_proj": {
+      "layer": 14,
+      "site": "k",
+      "t": 0.0,
+      "risk_norm": 0.0,
+      "conc": 0.16336153047732316,
+      "Einf": 0.0011944089054408448
+    }
+  },
+  "layer_view": {
+    "14": {
+      "base_model.model.model.layers.14.mlp.up_proj": {
+        "site": "up",
+        "t": 0.8129976992426243
+      },
+      "base_model.model.model.layers.14.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.7335376639204393
+      },
+      "base_model.model.model.layers.14.mlp.down_proj": {
+        "site": "down",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.14.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.14.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.14.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.14.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      }
+    },
+    "15": {
+      "base_model.model.model.layers.15.mlp.up_proj": {
+        "site": "up",
+        "t": 0.7235717332710954
+      },
+      "base_model.model.model.layers.15.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.7007285972024974
+      },
+      "base_model.model.model.layers.15.mlp.down_proj": {
+        "site": "down",
+        "t": 0.05170055028997815
+      },
+      "base_model.model.model.layers.15.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.15.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.15.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.15.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      }
+    },
+    "12": {
+      "base_model.model.model.layers.12.mlp.up_proj": {
+        "site": "up",
+        "t": 0.8040670570506432
+      },
+      "base_model.model.model.layers.12.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.62047731449069
+      },
+      "base_model.model.model.layers.12.mlp.down_proj": {
+        "site": "down",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.12.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.12.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.12.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.12.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      }
+    },
+    "1": {
+      "base_model.model.model.layers.1.mlp.down_proj": {
+        "site": "down",
+        "t": 0.852820759307394
+      },
+      "base_model.model.model.layers.1.mlp.up_proj": {
+        "site": "up",
+        "t": 0.4239953417816056
+      },
+      "base_model.model.model.layers.1.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.07666286945292178
+      },
+      "base_model.model.model.layers.1.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.1.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.1.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.1.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      }
+    },
+    "13": {
+      "base_model.model.model.layers.13.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.7217311024300797
+      },
+      "base_model.model.model.layers.13.mlp.up_proj": {
+        "site": "up",
+        "t": 0.6710380698353853
+      },
+      "base_model.model.model.layers.13.mlp.down_proj": {
+        "site": "down",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.13.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.13.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.13.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.13.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      }
+    },
+    "7": {
+      "base_model.model.model.layers.7.mlp.up_proj": {
+        "site": "up",
+        "t": 0.7783005199480901
+      },
+      "base_model.model.model.layers.7.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.2942002352889795
+      },
+      "base_model.model.model.layers.7.mlp.down_proj": {
+        "site": "down",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.7.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.7.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.7.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.7.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      }
+    },
+    "5": {
+      "base_model.model.model.layers.5.mlp.up_proj": {
+        "site": "up",
+        "t": 0.690164909922832
+      },
+      "base_model.model.model.layers.5.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.2331380993725579
+      },
+      "base_model.model.model.layers.5.mlp.down_proj": {
+        "site": "down",
+        "t": 0.02696106887579347
+      },
+      "base_model.model.model.layers.5.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.5.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.5.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.5.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      }
+    },
+    "4": {
+      "base_model.model.model.layers.4.mlp.up_proj": {
+        "site": "up",
+        "t": 0.636118019193088
+      },
+      "base_model.model.model.layers.4.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.49967339162172797
+      },
+      "base_model.model.model.layers.4.mlp.down_proj": {
+        "site": "down",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.4.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.4.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.4.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.4.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      }
+    },
+    "11": {
+      "base_model.model.model.layers.11.mlp.up_proj": {
+        "site": "up",
+        "t": 0.6347402283240952
+      },
+      "base_model.model.model.layers.11.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.5039493435609566
+      },
+      "base_model.model.model.layers.11.mlp.down_proj": {
+        "site": "down",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.11.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.11.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.11.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.11.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      }
+    },
+    "6": {
+      "base_model.model.model.layers.6.mlp.up_proj": {
+        "site": "up",
+        "t": 0.6530928346877478
+      },
+      "base_model.model.model.layers.6.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.5517768087168149
+      },
+      "base_model.model.model.layers.6.mlp.down_proj": {
+        "site": "down",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.6.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.6.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.6.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.6.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      }
+    },
+    "10": {
+      "base_model.model.model.layers.10.mlp.up_proj": {
+        "site": "up",
+        "t": 0.6537704992680213
+      },
+      "base_model.model.model.layers.10.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.44427349837398933
+      },
+      "base_model.model.model.layers.10.mlp.down_proj": {
+        "site": "down",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.10.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.10.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.10.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.10.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      }
+    },
+    "8": {
+      "base_model.model.model.layers.8.mlp.up_proj": {
+        "site": "up",
+        "t": 0.7084386947291339
+      },
+      "base_model.model.model.layers.8.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.22847834183504917
+      },
+      "base_model.model.model.layers.8.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.8.mlp.down_proj": {
+        "site": "down",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.8.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.8.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.8.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      }
+    },
+    "9": {
+      "base_model.model.model.layers.9.mlp.up_proj": {
+        "site": "up",
+        "t": 0.6274885868185597
+      },
+      "base_model.model.model.layers.9.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.5337633143074141
+      },
+      "base_model.model.model.layers.9.mlp.down_proj": {
+        "site": "down",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.9.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.9.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.9.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.9.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      }
+    },
+    "3": {
+      "base_model.model.model.layers.3.mlp.up_proj": {
+        "site": "up",
+        "t": 0.5976267038889456
+      },
+      "base_model.model.model.layers.3.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.2614555971813346
+      },
+      "base_model.model.model.layers.3.mlp.down_proj": {
+        "site": "down",
+        "t": 0.0019679547897706873
+      },
+      "base_model.model.model.layers.3.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.3.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.3.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.3.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      }
+    },
+    "0": {
+      "base_model.model.model.layers.0.mlp.down_proj": {
+        "site": "down",
+        "t": 0.5774169746344644
+      },
+      "base_model.model.model.layers.0.mlp.up_proj": {
+        "site": "up",
+        "t": 0.4680067424333691
+      },
+      "base_model.model.model.layers.0.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.12047786016271766
+      },
+      "base_model.model.model.layers.0.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.0.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.0.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.0.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      }
+    },
+    "2": {
+      "base_model.model.model.layers.2.mlp.up_proj": {
+        "site": "up",
+        "t": 0.5473159019958165
+      },
+      "base_model.model.model.layers.2.mlp.gate_proj": {
+        "site": "gate",
+        "t": 0.1944066721607684
+      },
+      "base_model.model.model.layers.2.mlp.down_proj": {
+        "site": "down",
+        "t": 0.018606340442403668
+      },
+      "base_model.model.model.layers.2.self_attn.v_proj": {
+        "site": "v",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.2.self_attn.o_proj": {
+        "site": "o",
+        "t": 0.0
+      },
+      "base_model.model.model.layers.2.self_attn.q_proj": {
+        "site": "q",
+        "t": 0.0013359830432677633
+      },
+      "base_model.model.model.layers.2.self_attn.k_proj": {
+        "site": "k",
+        "t": 0.0
+      }
+    }
+  }
+}