L3-NA-Aethora-15B

This is the NON-Abliterated VERSION and Experimental!!

The Skullery Presents L3-NA-Aethora-15B.

This is the NON-Abliterated VERSION and Experimental!!

Creator: Steelskull

Dataset: Aether-Lite-V1.2

Trained: 4 x A100 for 15 hours Using RsLora and DORA

About L3-NA-Aethora-15B:

L3 = Llama3 
NA = NON-ABLITERATED

L3-NA-Aethora-15B was crafted by using a modified DUS (Depth Up Scale) merge (originally used by @Elinas) by using passthrough merge to create a 15b model, with specific adjustments (zeroing) to 'o_proj' and 'down_proj', enhancing its efficiency and reducing perplexity. This created Meta-Llama-3-15b-Instruct.

Meta-Llama-3-15b-Instruct was then trained for 4 epochs using Rslora & DORA training methods on the Aether-Lite-V1.2 dataset, containing ~82000 high quality samples, designed to strike a fine balance between creativity, slop, and intelligence at about a 60/40 split

This model is trained on the L3 prompt format.

Quants:

Dataset Summary: (Filtered)

Filtered Phrases: GPTslop, Claudism's

• mrfakename/Pure-Dove-ShareGPT: Processed 3707, Removed 150
• mrfakename/Capybara-ShareGPT: Processed 13412, Removed 2594
• jondurbin/airoboros-3.2: Processed 54517, Removed 4192
• PJMixers/grimulkan_theory-of-mind-ShareGPT: Processed 533, Removed 6
• grimulkan/PIPPA-augmented-dedup: Processed 869, Removed 46
• grimulkan/LimaRP-augmented: Processed 790, Removed 14
• PJMixers/grimulkan_physical-reasoning-ShareGPT: Processed 895, Removed 4
• MinervaAI/Aesir-Preview: Processed 994, Removed 6
• Doctor-Shotgun/no-robots-sharegpt: Processed 9911, Removed 89

Deduplication Stats:

Starting row count: 85628, Final row count: 81960, Rows removed: 3668