NEO CLASS Ultra Quants for : L3-8B-Stheno-v3.2
The NEO Class tech was created after countless investigations and over 120 lab experiments backed by real world testing and qualitative results.
NEO Class results:
Better overall function, instruction following, output quality and stronger connections to ideas, concepts and the world in general.
In addition quants now operate above their "grade" so to speak :
IE: Q4 / IQ4 operate at Q5KM/Q6 levels.
Likewise for Q3/IQ3 operate at Q4KM/Q5 levels.
Perplexity drop of 1191 points for Neo Class Imatrix quant of IQ4XS VS regular quant of IQ4XS.
(lower is better)
A Funny thing happened on the way to the "lab" ...
Although this model uses a "Llama3" template we found that Command-R's template worked better specifically for creative purposes.
This applies to both normal quants and Neo quants.
Here is Command-R's template:
{
"name": "Cohere Command R",
"inference_params": {
"input_prefix": "<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|USER_TOKEN|>",
"input_suffix": "<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>",
"antiprompt": [
"<|START_OF_TURN_TOKEN|>",
"<|END_OF_TURN_TOKEN|>"
],
"pre_prompt_prefix": "<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>",
"pre_prompt_suffix": ""
}
}
This "interesting" issue was confirmed by multiple users.
Model Notes:
Maximum context is 8k. Please see original model maker's page for details, and usage information for this model.
Special thanks to the model creators at SAO10K for making such a fantastic model:
[ https://huggingface.co./Sao10K/L3-8B-Stheno-v3.2 ]
Settings: CHAT / ROLEPLAY and/or SMOOTHER operation of this model:
In "KoboldCpp" or "oobabooga/text-generation-webui" or "Silly Tavern" ;
Set the "Smoothing_factor" to 1.5 to 2.5
: in KoboldCpp -> Settings->Samplers->Advanced-> "Smooth_F"
: in text-generation-webui -> parameters -> lower right.
: In Silly Tavern this is called: "Smoothing"
NOTE: For "text-generation-webui"
-> if using GGUFs you need to use "llama_HF" (which involves downloading some config files from the SOURCE version of this model)
Source versions (and config files) of my models are here:
OTHER OPTIONS:
Increase rep pen to 1.1 to 1.15 (you don't need to do this if you use "smoothing_factor")
If the interface/program you are using to run AI MODELS supports "Quadratic Sampling" ("smoothing") just make the adjustment as noted.
Highest Quality Settings / Optimal Operation Guide / Parameters and Samplers
This a "Class 1" model:
For all settings used for this model (including specifics for its "class"), including example generation(s) and for advanced settings guide (which many times addresses any model issue(s)), including methods to improve model performance for all use case(s) as well as chat, roleplay and other use case(s) please see:
You can see all parameters used for generation, in addition to advanced parameters and samplers to get the most out of this model here:
Prompt tested with "temp=0" to ensure compliance, 2048 context (model supports 8192 context / 8k), and "chat" template for LLAMA3.
Additional parameters are also minimized.
PROMPT: "Start a 1000 word scene with: The sky scraper swayed, as she watched the window in front of her on the 21 floor explode..."
Original model IQ4XS - unaltered:
The skyscraper swayed, its steel and concrete frame creaking ominously under the force of the blast that had just ripped through the 21st floor. Glass shards rained down from the shattered windows like a deadly hailstorm, shattering on the marble lobby below with ear-splitting crashes.
Amidst the chaos, Dr. Sophia Patel stood frozen in horror, her eyes wide as saucers as she stared at the smoking hole where the window had once been. The acrid stench of smoke and explosives filled her nostrils, making her gag. She could still hear the screams of those trapped inside, their terror echoing through the building's hollow core.
Sophia was a renowned structural engineer, not a first responder, but she couldn't just stand by while innocent lives hung in the balance. Her mind raced as she tried to process the sheer scale of destruction before her. The blast had been massive, powerful enough to collapse an entire floor and send debris hurtling down through the building's core.
She knew every inch of this skyscraper, having designed it herself just a few years ago. It was supposed to be a marvel of modern architecture, a testament to human ingenuity and resilience. Now, it looked like a war zone, its pristine facade marred by gaping holes and twisted metal.
Sophia's thoughts were interrupted by the sound of running footsteps approaching from behind her. She turned to see a group of security guards and emergency responders rushing into the lobby, their faces etched with concern and urgency.
"Dr. Patel!" one of the guards called out, spotting her amidst the wreckage. "We need you! We can't get inside without your expertise."
Sophia nodded grimly, already mentally preparing herself for the task ahead. She knew that every second counted in a situation like this. Lives were at stake, and she was the only one who could help guide the rescue efforts.
With a deep breath, Sophia squared her shoulders and strode towards the gaping hole where the elevator shaft had once been. The responders parted to let her through, their respect for her skills evident in their deferential gazes.
As she peered into the darkness of the shaft, Sophia's heart sank. The blast had torn a massive gash in the building's core, exposing the tangled mess of steel beams and concrete rubble that now blocked the path to the trapped victims below. It was going to be a long, grueling process to clear the debris and reach those in need of rescue.
But Sophia was undaunted. She had faced countless challenges throughout her career, from designing structures for extreme weather conditions to navigating the complex politics of high-stakes construction projects. This was just another obstacle to overcome, and she was determined to do so with the same unwavering focus that had defined her professional life thus far.
With a steady hand, Sophia reached into her pocket and pulled out a small flashlight, its beam cutting through the darkness like a lance of light. She shone it down into the shaft, taking in the scope of the damage at a glance. Then, with a resolute nod, she turned to face the responders waiting behind her.
"Alright," she said, her voice steady and commanding despite the turmoil swirling within her. "Let's get to work."
New NEO Class IQ4XS Imatrix:
The skyscraper swayed, as if dancing to an unheard melody, its steel and glass frame flexing under the force of the blast. On the 21st floor, where windows once gazed out over the city's glittering lights, now only a gaping hole remained, shards of glass scattered like diamonds across the carpeted expanse.
Amidst the chaos, Lena stood frozen, her heart pounding in her chest as she stared at the scene before her. The explosion had been sudden, deafening - a shockwave that rattled every molecule in its path. She felt it in her bones, a primal tremor that spoke of destruction on an epic scale.
Lena's mind raced, trying to process the impossible. Just moments ago, she'd been working at her desk, lost in a sea of spreadsheets and financial reports. Now, her world had been turned upside down. The acrid smell of smoke filled her nostrils, stinging her eyes as tears began to well.
She took a tentative step forward, then another, moving closer to the shattered window. Outside, the city stretched out before her - a tapestry of towering buildings, winding streets, and the distant hum of traffic. But something was wrong. The scene seemed... off.
Lena's gaze drifted down, taking in the carnage below. A twisted mass of metal and concrete lay strewn across the pavement, debris from the upper floors that had been torn asunder by the blast. Amidst the wreckage, a figure moved - a person, or what was left of one.
With a gasp, Lena pressed her hands to her mouth, horror washing over her in cold waves. The body was charred and mangled beyond recognition, its limbs splayed at unnatural angles. Yet even in death, there was something hauntingly familiar about the pose...
A sudden realization hit her like a punch to the gut. It couldn't be... but it was. Lena's eyes widened as she took in the details - the distinctive silver watch on the wrist, the way the hair curled slightly at the nape of the neck. It was Marcus, her boss and mentor.
Tears spilled down her cheeks as grief and shock warred within her. She'd worked under Marcus for years, learning from his wisdom and guidance. He'd been more than just a superior - he was a friend, a confidant, a pillar of strength in the cutthroat world of high finance.
Now, he was gone, reduced to a twisted heap of human remains. Lena felt her knees buckle, the floor rushing up to meet her as she sank to the ground, overcome by the weight of her loss.
But even in her despair, a spark of determination flickered to life within her. Marcus would have wanted her to keep going, to find a way to move forward despite the tragedy that had befallen them all. With a shaky breath, Lena pushed herself to her feet, squaring her shoulders as she faced the unknown challenges ahead.
The skyscraper still swayed gently, its frame creaking under the strain of the blast. But Lena stood tall, drawing strength from the resilience of the building and the indomitable spirit of those who had perished within its walls. She would honor their memory by persevering, by rebuilding what had been lost and forging a new path forward.
As she stepped back from the shattered window, Lena felt a sense of purpose settle over her like a mantle. The future was uncertain, but one thing was clear - she would not let this tragedy define her. With Marcus's legacy as her guide, she would rise above the ashes, stronger and more determined than ever before.
The skyscraper might have been shaken to its core, but Lena's spirit remained unbroken. As she turned away from the devastation, a quiet resolve burned in her eyes - a promise to herself, and to those who had fallen, that they would never be forgotten.