I was just coming to post the opposite. My issue with v2 was that it's 'dumb' and didn't understand the story context, but v3 seems to be smarter in very initial testing.

I'm testing single-turn novel-style responses, the Cohere template for a system prompt, exl3, and temp-last sampling and a fairly low temperature; higher temps do go off the rails quick.

More testing to slowly come...

@yanno2mch

I made an exl3 here, forced 3bpw up_proj and 4bpw for all other layers. I can just fit 16K in 24GB: https://huggingface.co/Downtown-Case/jukofyork_command-r-35b-writer-v3-exl3-3.75bpw-hb6

It should have less quantization loss than a default Q3_K.

It wouldn't surprise me if it's slightly broken still - for now I'm just running some more tests on gemma-2:9b as I have a feeling the problems I'm getting with command-r are due to some bugs in the huggingface transformers tokenizer.

It wouldn't surprise me if it's slightly broken still - I have a feeling the problems I'm getting with command-r are due to some bugs in the huggingface transformers tokenizer.

Yeah it kinda did seem like it was two overlapping replies, either making sense but together not. That would explain why the replies also otherwise felt a little off. I'll look forward to the next iteration.

@jukofyork Regarding the command-r and command-r+ tokenizers in general, I've noticed there are like 10 different variants of them floating about with different sizes ranging from 12.8mb to 20mb.

Your model's tokenizer sha256sum is c69a7ea6c0927dfac8c349186ebcf0466a4723c21cbdb2e850cf559f0bee92b8 which matches the current main branch of CohereLabs/c4ai-command-r-v01 and CohereLabs/c4ai-command-r-plus

This is the same as Downtown-Case's exl3 quant (nice I'll download this!)

The original tokenizer had this checksum: 0af6e6fe50ce1bb5611b103482de6bac000c82e06898138d57f35af121aec772, but then a HF staff member updated it to add tool calling tokens. When I diff'd them a while ago, I saw 5 tokens added (all related to tool calling)

Command-R+ originally had a 16.5MB tokenizer SHA256: 9619890aebac311d644236f49462d7f8618ebef7c7020c52645ccb597434a3c9 for a while, but now it has the same one as command-r-v0

They also changed the EOS token from 6 to 255001

And I'm not sure if this would have an impact: "tie_word_embeddings": False

lol @ this comment:

Remove comment in RoPE saying it is the same as llama
# copied from transformers.models.llama.modeling_llama.LlamaRotaryEmbedding with Llama->Cohere

I've seen various quants with a 20mb tokenizer.json too, and the tokenizer_config.json has been changed a few times.
God knows what llama.cpp / gguf is doing to them.

I can't find it now, but I think I saw they migrated to FastTokenizer at some point. I suspect something went wrong at one point, because I was using command-r+ at launch in exllamav2 as my daily model for everything for a while, and never saw the Chinese/Russian characters printed. But now I get them locally (.gguf) and even using openrouter and the cohere api directly.

p.s. +1 for gemma-2-9b, that's a very unique model (not gemma-2-27b) and I still run it sometimes.

Oh crap, I've likely been using the same tokenizer as I downloaded when if first came out (I squirrel away models on a couple of 20TB drives to save redownloading them on my slow ADSL). I usually check if the JSON files have been updated and refresh them, but it never occurred to me the tokenizer file itself would change :((

p.s. +1 for gemma-2-9b, that's a very unique model (not gemma-2-27b) and I still run it sometimes.

Yeah, it seems to be the smallest model people have successfully fine-tuned creative writing models on and this particular fine-tune/merge hybrid people still seem to like:

I'm pretty sure I'll soon have the Control Adapters stuff working as well as the Control Vectors:

• After a lot of hours working out the maths I've actually got something now that is almost exactly the multiplicative "partner" to the way I found to calculate the (additive) Control Vectors (ie: now actually parameterised as an Eigen-decompostion of a symmetric square matrix).
• I think I've found at last what causes the models to "drift" and screw up the non-writing ability too. I can't seem to find a good regulariser to avoid this: they all just end up turning my rank-64 adapter into having an effective-rank of ~1.5 and totally ruin it... BUT: by adding ~10% random data (eg: 50/50 webcrawl and instruct data) to the positive and negative classes, I can get a pretty similar effect and avoid the "drift" by insisting each "side" of the Control Adapter learn directions that don't screw up non-writing tasks that it would never see otherwise.

I've only got this working the last couple of days so pretty excited to see what comes out! The very first rank-16 version of command-r:35b and qwq:32b looked so promising, so fingers crossed!!! :D

Do any of you have any recommendations for other smallish models I can try? There are so many different mistral models in the 12-24B range, I've no idea which might be a good target?

I don't want to deal with gemma-3 yet as it needs two sets of RoPE cos/sin values passing through it and my forked version of qlora-pipe would need a fair bit of reworking to add these.

I'm particularly interested in smallish "smart" models that don't have great prose (but not terrible prose...).

It would be interesting to see if I can keep the "smarts" and just effect the prose...

(I have a feeling that the mixing @yano2mch is seeing in this model is because I separated the paragraphs using double newlines, whereas before [and again now] I separated using the EOS tokens).

(I have a feeling that the mixing @yano2mch is seeing in this model is because I separated the paragraphs using double newlines, whereas before [and again now] I separated using the EOS tokens).

Maybe. It kinda feels more like one sentence is written by someone and then the next sentence is written by someone else (i am harmed by sunlight, i will wait outside, it is light out right now), that or the logic/consistency doesn't fit in that instance for some reason (or i'm being too picky, that's possible). The vampire one i mentioned especially stands out. I'll post a bit of the output here to show what i mean.

Here's the important parts. Snipped out unnecessary flirting. Formatting is easier with BBCode. If you want the unaltered logs i can provide (which is mostly the intro and him rolling a rare vampire).

"My statistics, as you requested. I am a vampire, one of the most powerful of my kind. I possess enhanced strength, agility, and reflexes. I do not tire or grow bored, and my senses are far beyond those of a normal human. My hearing is especially keen. Because I am a vampire, I cannot withstand sunlight, and the holy symbols of certain religious orders have power over me." A frown creases her brow slightly. "However, you will find that I am a loyal companion. I will not betray you unless you abuse me." A small, enigmatic smile. "But that won't be a problem, will it? I can't wait to help you in your adventures. And who knows, perhaps you'll help me as well."

She steps back, giving you a deep,courteous curtsey. "It's nice to finally meet you, master." She gives you a wink. "Shall we begin?"

Yano: "A vampire. Not a big surprise based on your outfit..." he touches her cheek and murrs. "Can i summon and unsummon you?" he asks, and glances at the door, it is likely daytime right now. "I'm not sure we can safely leave if you can't handle sunlight. Sounds like we'll be taking night jobs then."

-snip-

Gacha Isekai: -snip-

She reaches forward and places her hands on your shoulders and leans in close. "Now, shall we go and see what's outside?"

Yano: "What happens when you are unsummoned? Are you nearby? Do you sleep as an object? or do you cease to be until called?" he asks.

He moves to the door and opens it a crack to look outside to see how bright it is, it might be safe to take her outside.

Gacha Isekai: "When you unsummon me, I remain close by, in a dormant state. I can sense the world around me but I cannot act or move until you summon me once more. I do not 'sleep' per say, I simply await your call. If you wish, I can wait outside for you while you tend to whatever business you have," Lamia suggests. "I can protect myself from most threats, and as the sun is out, I would not be of much use to you in any case." She steps back from you, a cautious expression on her face.

If you want the unaltered logs i can provide

Do you have an initial prompt I can try for comparison, for simple testing?

I ask because:

Do any of you have any recommendations for other smallish models I can try?

Maybe, but I feel like my own testing is pretty limited in scope, so I'd like to see how other's prompts do.

If you want the unaltered logs i can provide

Do you have an initial prompt I can try for comparison, for simple testing?

Initial prompt I used came from Gatcha Isekai, download the Json file (or at the bottom view the card) and you can probably build the overhead for the initial prompt if you don't use SillyTavern or TavernAI. Actually with oobadooba's webui also reads char cards. So that would be easier to get an initial setup.

https://char-archive.evulid.cc/#/chub/numb_action_7988/character/gacha-isekai-concept-765dec3d4f05

Makes me wish there was a PM/DM system here or the like.

Oh crap, I've likely been using the same tokenizer as I downloaded when if first came out

That's what I've been doing. Either that, or I downloaded an old gguf. Just made a Q6_K_M of command-r-v01 and so far no Cyrillic characters.
Your tokenizer does match the correct one though (sha256sum) and I re-ran my tokdiff

Loading tokenizers...
✅ Tokenizers loaded successfully!

============================================================
📊 TOKENIZER COMPARISON
============================================================
        Property             Command R Writer v3                   Command R v01
      Model Name         command-r-35b-writer-v3              c4ai-command-r-v01
      Vocab Size                          255029                          255029
Model Max Length 1000000000000000019884624838656 1000000000000000019884624838656
 Tokenizer Class             CohereTokenizerFast             CohereTokenizerFast

============================================================
🔤 SPECIAL TOKENS
============================================================
Token Type                Command R Writer v3                      Command R v01
       PAD                      <PAD> (ID: 0)                      <PAD> (ID: 0)
       BOS                <BOS_TOKEN> (ID: 5)                <BOS_TOKEN> (ID: 5)
       EOS <|END_OF_TURN_TOKEN|> (ID: 255001) <|END_OF_TURN_TOKEN|> (ID: 255001)
       UNK                    None (ID: None)                    None (ID: None)

============================================================
🔍 TOKENIZATION EXAMPLES
============================================================

1. Input: 'Hello, world!'
----------------------------------------
Command R Writer v3: 4 tokens
  Tokens: ['Hello', ',', 'Ġworld', '!']
  IDs: [28339, 19, 3845, 8]

Command R v01: 4 tokens
  Tokens: ['Hello', ',', 'Ġworld', '!']
  IDs: [28339, 19, 3845, 8]

...<omitted all the other tests for brevity>
============================================================
📈 VOCABULARY ANALYSIS
============================================================
Shared tokens: 255,029
Unique to Command R Writer v3: 0
Unique to Command R v01: 0
Overlap percentage: 100.00%

============================================================
⚡ EFFICIENCY TEST
============================================================
Long text (830 chars):
  Command R Writer v3: 101 tokens (ratio: 0.122)
  Command R v01: 101 tokens (ratio: 0.122)

it seems to be the smallest model people have successfully fine-tuned creative writing models

Yeah, it also responds very well to your control vectors. I'll have to try that DarkestMuse model

There are so many different mistral models in the 12-24B range, I've no idea which might be a good target?

Other than gemma-2-9b, I don't really use models in that range, but I keep reading that Nemo-12b and Mistral-Small-22b are good.

Most of the "writing" finetunes are probably SillyTavern/Roleplay traiend on synthetic datasets.

There are so many different mistral models in the 12-24B range, I've no idea which might be a good target?

Other than gemma-2-9b, I don't really use models in that range, but I keep reading that Nemo-12b and Mistral-Small-22b are good.

Most of the "writing" finetunes are probably SillyTavern/Roleplay trained on synthetic datasets.

I've been tinkering with different models since about 6 months ago. The best model to use, is the biggest one you can load and use that you're happy with (Use your RAM as your ultimate determination, probably 8-16Gb less than your max RAM). Even if you get slow output (a handful or 1 token a second) isn't bad, i did a lot of RPing where you get replies every 15 minutes. The more VRAM you have the faster the output will ultimately be.

So far for 8B The Trifecta does a pretty good job for almost everything i've thrown at it, including making lyrics for music and then passing to Lyricsintosong.ai. Does summarizing okay, writes halfway decently. The 20-24B model didn't seem to do any better in my mind.

But like a lot of 8B-16B models, it's better for one-on-one or not giving it anything too complicated. 16-24B models can generally do multiple characters, but i wouldn't give it too many.

33B models are where it starts getting good. 50B models and above could do roleplay-within-a-roleplay. 70B models are what i usually use if i can get it.

There's a 24B and 70B models called Omega Directive that i'm finding to be my current fallback. After that Anubis Lemonade 70B. Had nice conversations of writing styles and what LLMs are good for with the 8B Chaos model.

Ultimately depending on what you're looking for, try a model, give it a day or two with inputs and outputs and see if you're happy.

https://huggingface.co/mradermacher/Trifecta-L3-8b-i1-GGUF
https://huggingface.co/mradermacher/MS3.2-The-Omega-Directive-24B-Unslop-v2.1-GGUF
https://huggingface.co/ReadyArt/L3.3-The-Omega-Directive-70B-Unslop-v2.1
https://huggingface.co/mradermacher/AnubisLemonade-70B-v1-i1-GGUF
https://huggingface.co/Lewdiculous/Chaos_RP_l3_8B-GGUF-IQ-Imatrix

edit: As a note, Space Wars was pretty decent as a 24B model. But if you're range is up to 24B, try the Omega Directive :)

https://huggingface.co/mradermacher/Space-Wars-24B-v1.00a-i1-GGUF

I honestly can't tell a good small model from a bad small model, but if you want a "slop-generator" the inverse model trained on gemma-9b is amazing! :D

Using temperature = 0, lora-scale = 1:

<start_of_turn>user
Write me a fantasy story<end_of_turn>
<start_of_turn>model
The wind howled through the skeletal branches of the Whispering Woods, carrying with it the scent of damp earth and ancient magic. Elara, a young woman with eyes like molten gold and hair the color of raven feathers, pressed deeper into the forest, her heart pounding against her ribs like a trapped bird. She clutched the worn leather-bound book tighter, its pages whispering secrets only she could hear.

Elara was a Whisperer, one of the few remaining guardians of the ancient language of the forest. This language, spoken only by the trees themselves, held the key to unlocking the secrets of the world, and Elara had been chosen to decipher its mysteries. But her task was fraught with danger. The Shadow Blight, a creeping darkness that devoured all life in its path, was spreading across the land, and only the Whisperers could stop it.

As she ventured deeper, the trees grew denser, their branches intertwining overhead to form a canopy that blocked out the sun. The air grew heavy with an oppressive silence, broken only by the rustling of leaves and the distant caw of a raven. Elara felt a prickle of unease crawl up her spine. She wasn't alone.

A low growl echoed through the trees, sending shivers down her spine. From the shadows emerged a creature unlike any she had ever seen - a hulking beast with eyes like burning embers and fur as black as night. Its fangs dripped with venom, and its breath reeked of decay.

Elara knew this was a Shade, a creature born from the Shadow Blight, tasked with hunting down the Whisperers. Fear threatened to consume her, but she remembered her purpose. She had to protect the forest, even if it meant sacrificing herself.

Taking a deep breath, Elara opened the ancient book and began to speak. The words flowed from her lips, ancient and powerful, weaving a tapestry of light and sound that pulsed with the rhythm of the forest. The Shade snarled, its eyes flickering with confusion and pain as the light touched its form.

But the Shade was strong, fueled by the darkness that consumed the land. It lunged at Elara, its claws raking across her arm, drawing blood. Pain shot through her, but she refused to falter. She continued to speak, her voice growing stronger, her words resonating with the very essence of the forest.

Slowly, the Shade began to shrink, its form flickering like a dying flame. With a final, anguished cry, it dissolved into nothingness, leaving behind only a wisp of smoke and the scent of decay.

Elara collapsed to her knees, her body trembling with exhaustion. But she had won. For now. The Shadow Blight still loomed, a constant threat to the forest and all its inhabitants. But Elara knew she would continue to fight, to protect the ancient language and the secrets it held. For she was a Whisperer, and the fate of the world rested on her shoulders.

Using temperature = 0, lora-scale = 2:

<start_of_turn>user
Write me a fantasy story<end_of_turn>
<start_of_turn>model
The wind howled through the ancient trees of Whispering Woods, carrying with it the scent of pine needles and damp earth. A young girl named Elara, no older than ten, stood at the edge of the forest, her eyes wide with wonder and fear. She had heard tales of the creatures that lived within its depths - creatures of magic and myth, both beautiful and terrifying. But today, she was determined to find them.

She ventured deeper into the woods, following a narrow path that wound its way through the trees. As she walked, she noticed something unusual - the air shimmered and danced around her, as if alive. A voice whispered in her ear, soft and sweet, "Hello, little one. What brings you to our home?"

Elara turned to see a creature unlike anything she had ever imagined. It was tall and slender, with skin that shimmered like moonlight on water. Its eyes glowed with an inner light, and its voice echoed with the wisdom of ages.

"I'm Elara," she said, her voice barely above a whisper. "I've come to see the creatures of magic."

The creature smiled, its eyes twinkling with amusement. "Ah, so you seek the Fae Folk. But be warned, little one, not all of us are friendly."

Elara nodded, her heart pounding in her chest. "I know. But I want to see them anyway."

The creature sighed, its voice heavy with sadness. "Very well. Follow me."

They walked deeper into the woods, until they reached a clearing bathed in moonlight. In the center of the clearing stood a group of creatures, their forms shifting and changing like the shadows themselves. Some were beautiful, with wings of light and eyes that sparkled with mischief. Others were grotesque, with twisted limbs and faces that seemed to leer at Elara with malice.

Elara gasped, her eyes wide with wonder and fear. But the creature beside her placed a calming hand on her shoulder. "Do not be afraid, little one. These are the Fae Folk. Some are good, some are bad. But all are powerful."

As they watched, the Fae Folk began to dance, their movements fluid and graceful. They spun and twirled, their laughter echoing through the trees. And then, suddenly, the laughter stopped. A hush fell over the clearing, and the Fae Folk turned to face Elara, their eyes glowing with an intensity that made her shiver.

One of the Fae Folk stepped forward, its form shifting and changing until it took on the shape of a beautiful woman. Her eyes were filled with sorrow, and her voice trembled as she spoke. "We have been watching you, little one. We have seen your courage, your determination. And we know that you are destined for great things."

Elara looked at her, her heart pounding in her chest. "What do you mean?"

The Fae Folk smiled, a sad smile that tugged at Elara's heart. "You are one of us, little one. You are a Fae Folk, too. And you have a destiny to fulfill."

Elara stared at her, her mind reeling. A Fae Folk? But she didn't look like them. She didn't feel like them.

The Fae Folk placed a hand on Elara's shoulder, her touch warm and comforting. "You may not look like us, little one. But you are one of us. And you have a destiny to fulfill. A destiny that will change the world."

And then, with a flash of light, the Fae Folk disappeared, leaving Elara alone in the clearing. She stood there for a long time, staring at the spot where they had been, her mind racing. A Fae Folk? A destiny? What did it all mean?

But then, as the first rays of dawn broke through the trees, she knew what she had to do. She had to find her destiny. She had to become the Fae Folk she was meant to be.

And so, Elara turned and walked out of the woods, her heart filled with hope and determination. She didn't know what lay ahead, but she knew that she was ready to face it. She was ready to become the Fae Folk she was meant to be.

In those 2 stories it's pulled in just about every slop-phrase going!

It's not finished training properly for about 10 hours, but looks interesting... Keen to try again on command-r, but might retry on command-r:32b this time.

The proper (non-inverse) model using temperature = 0, lora-scale = 1:

<start_of_turn>user
Write me a fantasy story<end_of_turn>
<start_of_turn>model
The wind whispered secrets through the silver leaves of the Whisperwood, secrets only Elara, the last Whisperer, could understand. She knelt beside the ancient oak, its gnarled roots snaking through the forest floor like sleeping serpents. The oak was the heart of the Whisperwood, its wisdom echoing through the rustling leaves, its magic woven into the very air.

Elara pressed her hand against the rough bark, feeling the pulse of the forest thrumming beneath her fingertips. The Whisperwood was dying. A creeping blight, a darkness that choked the life from the trees, was spreading like a stain across the land. The whispers, once vibrant and full of life, were now choked with fear and despair.

"What can I do?" Elara whispered, her voice a mere breath in the stillness of the wood.

The oak rumbled, its voice a deep, resonant echo in her mind. "The Shadowblight is fueled by fear. You must find the source, the heart of the darkness, and banish it."

Elara's heart sank. The heart of the Shadowblight was said to be guarded by the Nightwalker, a creature of pure darkness, a nightmare given form. No Whisperer had ever faced the Nightwalker and lived to tell the tale.

But Elara had no choice. The Whisperwood was her home, her lifeblood. She had to save it, even if it meant facing her greatest fear.

She rose, her eyes fixed on the darkening forest. The whispers grew fainter, the air thick with a cloying, oppressive scent. The blight was spreading.

Guided by the oak's whispers, Elara ventured deeper into the Whisperwood. The trees grew twisted and gnarled, their leaves withered and black. The air was heavy with the stench of decay.

Days turned into weeks, weeks into months. Elara battled through the blighted forest, her strength waning, her hope dwindling. She fought off creatures born of the Shadowblight, their eyes burning with malevolent fire. She faced her own fears, the whispers of doubt and despair gnawing at her resolve.

Finally, she reached a clearing. In the center stood a towering obsidian obelisk, its surface pulsing with a sickly green light. The air crackled with dark energy. This was the heart of the Shadowblight.

And guarding it was the Nightwalker.

It was a creature of nightmares, a towering silhouette wreathed in shadows. Its eyes glowed with an infernal light, its claws dripping with venom.

Elara drew her sword, its blade shimmering with the last vestiges of the Whisperwood's magic. She knew this was her final stand.

"I am Elara, the last Whisperer," she declared, her voice trembling but resolute. "I have come to banish you and your darkness from this land."

The Nightwalker let out a chilling shriek, its voice a chorus of tormented souls. It lunged at Elara, its claws slashing through the air.

Elara dodged, her sword flashing. She fought with the fury of a cornered animal, her every move fueled by the love for her home, the hope for its survival.

The battle raged, a whirlwind of steel and shadow. Elara's strength was failing, but she refused to give up. She remembered the oak's words, the whispers of the forest, the hope that still flickered within her.

With a final, desperate lunge, Elara thrust her sword into the Nightwalker's heart. The creature let out a deafening roar, its form dissolving into a cloud of black smoke.

The obelisk shattered, its green light fading. The blight receded, the air clearing, the whispers returning, faint at first, then growing stronger, filled with relief and gratitude.

Elara collapsed, her body wracked with exhaustion. But she had won. The Whisperwood was saved.

As the sun rose, casting its golden light upon the forest, Elara knew her journey was far from over. The Whisperwood needed her, and she would be its voice, its protector, its hope.

The proper (non-inverse) model using temperature = 0, lora-scale = 1.5:

<start_of_turn>user
Write me a fantasy story<end_of_turn>
<start_of_turn>model
The wind whipped Elara's hair across her face as she clung to the cliff face. Below, the sea churned, a frothing mass of grey and white. The storm had come on suddenly, a wall of wind and rain that had driven her from the safety of the cave. Now, she was trapped, clinging to the rock with fingers that were numb with cold.

She had been following the whispers, the rumours of a hidden island, a place where the veil between worlds was thin. A place where magic flowed freely. She had travelled for months, following the whispers, her heart filled with hope. But the storm had caught her off guard, and now she was paying the price.

Elara closed her eyes, trying to focus on the whispers. They were faint, almost drowned out by the roar of the wind, but she could still hear them. They spoke of a hidden path, a way to reach the island. She had to find it. She had to reach the island.

She opened her eyes and scanned the cliff face. The rain was coming down in sheets, making it hard to see. But then, she saw it. A small indentation in the rock, barely visible in the storm. It was a handhold, a way up.

Elara took a deep breath and reached for it. Her fingers found the rough surface of the rock. She pulled herself up, her muscles screaming in protest. The wind buffeted her, trying to knock her off the cliff. But she held on, her eyes fixed on the path above.

Slowly, painfully, she climbed. The rain soaked through her clothes, chilling her to the bone. But she kept climbing, driven by the whispers, by the hope that she would reach the island.

Finally, she reached the top of the cliff. She collapsed on the ground, her body trembling with exhaustion. She had made it. She had reached the island.

The storm was still raging, but the wind had died down. Elara looked around her. The island was small, a rocky outcrop surrounded by churning water. But it was beautiful. The air was thick with the scent of salt and seaweed. The sky was a swirling mass of grey and purple.

Elara stood up and walked towards the centre of the island. She could feel the magic here, a tingling sensation that ran through her body. She closed her eyes and took a deep breath. The whispers were louder now, clearer. They spoke of a hidden cave, a place of power.

Elara followed the whispers, her heart pounding with anticipation. She walked through a grove of trees, their branches twisted and gnarled. She came to a clearing, and there, in the centre, was a cave.

The cave was dark and damp. Elara stepped inside, her eyes adjusting to the gloom. The air was thick with the smell of earth and stone. She could feel the magic here, stronger than ever.

She walked deeper into the cave, her footsteps echoing in the silence. The walls were covered in strange symbols, glowing faintly in the darkness. She reached a chamber, and in the centre, there was a pool of water. The water was black as night, but it shimmered with an inner light.

Elara knelt down and cupped her hands in the water. It was cold, but it felt alive. She could feel the magic flowing through it, connecting her to something ancient and powerful.

She closed her eyes and took a deep breath. She had found it. The island of whispers. The place where magic flowed freely. And she was ready to embrace it.

(it starts to break down with lora-scale = 2)

Seems way darker! :D

<start_of_turn>user

Varis adjusted the noose, its hemp fibers grinding beneath his calluses. "Last chance," he said, voice like gravel dragged through mud. "Confess, and your soul stays your own."

Jurl laughed - a wet, gurgling sound. “You’re knee-deep in it, Coldwater." The thing inside him twisted the boy's lips into a grin too wide for his face. "The Great Wolf's howlin' again. The Dead's Gate's rusted through…"

Turn this into the opening chapter of a Grimdark trilogy.<end_of_turn>
<start_of_turn>model

The Wolf's Hunger

Chapter 1: The Hangman's Grace

The wind howled through the skeletal branches of the gallows tree, a mournful dirge for the condemned. Varis Coldwater, the hangman of Blackhaven, adjusted the noose, its hemp fibers grinding beneath his calluses. The rope was old, its fibres worn thin by countless executions, but it would do. It always did.

"Last chance," he said, his voice like gravel dragged through mud. "Confess, and your soul stays your own."

Jurl, the boy condemned, laughed. It was a wet, gurgling sound, the kind that makes the blood run cold. The thing inside him twisted the boy's lips into a grin too wide for his face.

"You're knee-deep in it, Coldwater," Jurl said, his voice a hollow rasp. "The Great Wolf's howlin' again. The Dead's Gate's rusted through…"

Varis ignored the boy's words. He knew what the boy was talking about. The Dead's Gate, the barrier between the world of the living and the realm of the dead, had been weakening for years. The Great Wolf, a monstrous entity of shadow and bone, was stirring in the darkness beyond. It was only a matter of time before it broke through, and when it did, Blackhaven, and all of the world, would be consumed.

But Varis wasn't a prophet. He was a hangman. His job was to execute the condemned, not to save the world. He had no power to stop the coming darkness. He could only do his duty.

He stepped forward and placed the noose around Jurl's neck. The boy's eyes were wide with fear, but there was something else there too, something dark and twisted. Varis had seen it before in the eyes of the condemned, a glimpse of the monstrous thing that lurked within them, waiting to be unleashed.

"Say your prayers, boy," Varis said. "You'll need them."

He pulled the lever, and the trapdoor beneath Jurl's feet opened. The boy's body went limp, his head snapping back as the noose tightened around his neck. His eyes stared blankly at the sky, his mouth open in a silent scream.

Varis watched as the boy's life ebbed away. He felt nothing. He had seen too many deaths to feel anything anymore. He was a machine, a tool of the state. He was the hangman of Blackhaven, and he would continue to do his duty until the day he died.

But as he watched Jurl's body swing in the wind, a strange feeling crept into Varis's heart. It was a feeling of dread, of foreboding. He couldn't shake the feeling that something was wrong. Something was coming.

And it was hungry.

<start_of_turn>user

Varis adjusted the noose, its hemp fibers grinding beneath his calluses. "Last chance," he said, voice like gravel dragged through mud. "Confess, and your soul stays your own."

Jurl laughed - a wet, gurgling sound. “You’re knee-deep in it, Coldwater." The thing inside him twisted the boy's lips into a grin too wide for his face. "The Great Wolf's howlin' again. The Dead's Gate's rusted through…"

Turn this into the opening chapter of a Grimdark trilogy.<end_of_turn>
<start_of_turn>model

Chapter 1: The Howling Wolf

The air hung heavy with the stench of fear and damp earth. Varis, known as Coldwater in these parts, adjusted the noose, its hemp fibers grinding beneath his calluses. He wasn't a man given to sentimentality, but even he felt a pang of pity for the boy before him. Jurl, barely a man, his face pale and drawn, eyes wide with terror.

"Last chance," Varis said, his voice like gravel dragged through mud. "Confess, and your soul stays your own."

Jurl laughed - a wet, gurgling sound that echoed unnaturally in the cavernous space beneath the crumbling city walls. "You're knee-deep in it, Coldwater." The thing inside him twisted the boy's lips into a grin too wide for his face. "The Great Wolf's howlin' again. The Dead's Gate's rusted through…"

Varis felt a chill crawl down his spine. The Great Wolf. A name whispered in hushed tones, a legend born from nightmares. A creature of pure shadow and malice, said to roam the blighted lands beyond the city walls, feeding on the souls of the damned. And the Dead's Gate…that was a different kind of terror. A gateway to the realm of the dead, said to be sealed by ancient magic, now weakened, its rusted hinges groaning under the weight of the encroaching darkness.

He had seen the signs. The whispers in the wind, the shadows growing longer, the nightmares seeping into the waking world. He had seen the fear in the eyes of the city guard, the desperation in the faces of the citizens. And now, here he was, facing a boy possessed by something ancient and evil, a harbinger of the coming storm.

"What do you want?" Varis asked, his voice barely above a whisper.

Jurl's grin widened, revealing teeth that were too sharp, too long. "We want what we've always wanted," he hissed. "To break free. To feast on the living. To drown this world in darkness."

Varis drew a deep breath, steeling himself for the fight ahead. He knew what he had to do. He had to stop the boy, even if it meant sacrificing his own soul.

He tightened the noose, the hemp fibers biting into Jurl's neck. "Then let's dance," he said, his voice cold and hard. "Let's dance with the devil."

The boy's eyes widened, a flicker of fear crossing his face. But it was quickly replaced by a manic grin. "You're already dancing, Coldwater," he said. "You're dancing with the wolf."

And as the boy's body convulsed, a shadow detached itself from him, taking the form of a wolf, its eyes glowing with an unholy light. It snarled, its teeth bared, and lunged at Varis.

The fight was brutal, a dance of death beneath the crumbling city walls. Varis fought with the desperation of a man facing his own mortality, his every move fueled by the knowledge that the fate of the world hung in the balance.

But even as he fought, he knew that this was only the beginning. The Great Wolf was loose, and the Dead's Gate was open. The darkness was coming, and there was nothing he could do to stop it.
To be continued...

<start_of_turn>user

Varis adjusted the noose, its hemp fibers grinding beneath his calluses. "Last chance," he said, voice like gravel dragged through mud. "Confess, and your soul stays your own."

Jurl laughed - a wet, gurgling sound. “You’re knee-deep in it, Coldwater." The thing inside him twisted the boy's lips into a grin too wide for his face. "The Great Wolf's howlin' again. The Dead's Gate's rusted through…"

Turn this into the opening chapter of a Grimdark trilogy.<end_of_turn>
<start_of_turn>model

Chapter 1: The Wolf's Hunger

The air hung thick with the stench of brine and decay, a miasma clinging to the cobblestones of Saltwind like a shroud. Above, the sky was the colour of old bruises, bruised further by the perpetual drizzle that seemed to seep into the very marrow of the city. Varis Coldwater, his face a roadmap of past battles and hard living, adjusted the hemp noose, its rough fibers biting into his calloused fingers.

"Last chance," he rasped, his voice a gravelly echo in the damp alleyway. "Confess, and your soul stays your own."

Jurl, a boy barely past his teens, stared back with eyes that burned with an unnatural light. The flesh around his mouth stretched taut, pulled into a grotesque grin by something unseen, something that twisted his features into a mockery of humanity.

"You're knee-deep in it, Coldwater," Jurl croaked, his voice a wet, gurgling sound that sent shivers down Varis' spine. "The Great Wolf's howlin' again. The Dead's Gate's rusted through…"

Varis felt a chill crawl up his spine, colder than the perpetual drizzle. He knew the boy spoke the truth. The whispers had been growing louder, the nightmares more vivid. The Dead's Gate, the ancient barrier that held back the horrors of the Abyss, was weakening. And the Great Wolf, a monstrous entity of pure malice, was stirring.

He had come to Saltwind seeking redemption, a chance to atone for the sins of his past. But the city, a festering wound on the face of the world, seemed determined to drag him further into darkness. Jurl, a pawn in a game far larger than himself, was just the first piece on a bloody chessboard.

Varis tightened his grip on the noose, his heart a leaden weight in his chest. He knew what he had to do. He had to stop the boy, even if it meant sacrificing his own soul.

"Then confess," Varis growled, his voice barely a whisper. "Tell me what you know. Tell me how to stop it."

Jurl's grin widened, revealing teeth that were too sharp, too long. "You can't stop it, Coldwater," he hissed. "The Wolf's hunger is insatiable. And soon, it will feast on us all."

The boy's laughter echoed through the alleyway, a chilling prelude to the coming storm. Varis knew then that the fight for Saltwind, for the world itself, had already begun. And he was caught in its bloody heart.

I flipped a coin to decide on the ordering above. All with temperature = 0.

• One is the original gemma:9b.
• One is the finetuned model with lora-scale = 1.5.
• One is the inverted model with lora-scale = 1.5.

Is it obvious which is which?

I honestly can't tell a good small model from a bad small model

• Inconsistencies (but not so much due to out of context)
• Doesn't follow instructions well
• outputting incomprehensible garbage (bad quant, corrupted model, bad training)
• Heavy reuse of phrases. (In one promising model, i got several replies of on par with 'her avatar broke apart in a blue digital matrix of fine fractals before reforming into a sharper image'... word for word... at the beginning of every reply)
• Too many line breaks or too few line breaks (wall of text, or every 8 words a newline without reason).
  A number of models will adjust the linebreaks based on the history so if it self-adjusts that isn't so bad, but you HAVE to fix all errors as you find them.
• Missing words. (Several cases I've had missing identifiers and pronouns. So he/she/it/the/they are missing. The sentence still makes sense, but over time it gets worse)

Also tried a SOAR model, which i didn't apparently set up right and all the replies were very very basic. Like minimally. But that's more my own fault than it's fault. Author did admit it was probably bad for RPing so i haven't tried any SOAR models since.

If the model is functional, it comes down to trying it and getting a feel for it. I have a list of models when i first started, with one or two recommended ones.

One thing about those old models is that they completely fall apart at longer context, or at best simply ignore most of it. Even with the sloppiness, newer stuff like Gemma3 blow them away in that respect.

Hence my fascination with the Bytedance model: it's freaking smart over 32K, though I still have mixed feelings about its prose. So I'm going back to testing it with a half base model merge: https://huggingface.co/Downtown-Case/Seed-OSS-36B-Base-Instruct-Karcher-Merge

...But I think it fits the category of 'smart, just needs better prose'

Oh, actually, there is one 'small model' exception I remember:

https://huggingface.co/models?sort=modified&search=tifa+deepsex

Contrary to the name, the 7Bs and 14Bs are more story than horny tuned. They're apparently multi-stage, long context full finetunes on a big literature corpus, and in spite of the controversy around them, I remember the 14B holding together shockingly well (in English) for the size.

The original uploads are GGUFs, but HF conversions are floating around there somewhere.

One thing about those old models is that they completely fall apart at longer context, or at best simply ignore most of it. Even with the sloppiness, newer stuff like Gemma3 blow them away in that respect.

Mhmm. When i first started i had a 4Gb video card, and 32Gb Ram. Which made doing 12B models difficult. With 128Gb ram and 8Gb VRam it's a bit better now.

I also noticed when you pushed too much on a lot of the models, they start outputting what you gave it rather than analyzing content. Quite limiting and an annoyance.

Hence my fascination with the Bytedance model-snip- OSS-36B-Base -snip-

Yeah i rarely touch anything under 24B anymore after a slight hardware upgrade. I'll certainly look at Bytedance too. Usually i prefer 30B or larger models as a minimum.

Contrary to the name, the 7Bs and 14Bs are more story than horny tuned. They're apparently multi-stage, long context full finetunes on a big literature corpus, and in spite of the controversy around them, I remember the 14B holding together shockingly well (in English) for the size.

I notice certain phrases get heavily repeated a lot when you get models in a certain position. 'Don't think for one minute that ___' being one i have seen too many times.

Rather than story generation i tend to RP; But my sheet of tested models the 'horny' was just a way to determine how censored/uncensored it was and less for randy RP's (though i do enjoy my ERPs :P).

I'd usually ask it to generate a short story with 2-3 things it really didn't like and see if it barfs or not, i didn't care about the output. (some outputs were okay, others were.. meh...). But one thing for sure the smaller models rush to a conclusion (generally all in a singular post), while larger models actually let the scene go on for a few posts, mimicking a proper ERP.

Though i had some tests where i have it rewrite some particularly short and less than detailed stories and got some pretty darn good outputs.

If anyone's interested...
https://forum.allporncomix.com/threads/yano2mch-experiments-in-story-generation.5332/post-151707

It's not finished training yet, but it appears to have at least done something:

gemma-2-writer-9b-lora

https://pastebin.com/tG0NbJGV

gemma-2-inverse-writer-9b-lora

https://pastebin.com/393U4dyh