Another banger!
Been flip-flopping between this and Wissenschaft for a project I'm currently doing, but I think the extra training done here really helped with it overall, so I gotta thank you for that! Your simple but effective Gutenberg versions have really help unlock some of these model's potential I feel.
Any chance of seeing this specific recipe with the new 22B Mistral? My brief initial testing with the new Instruct was kinda lukewarm, tbh. Maybe it just doesn't like the same settings I used with Nemo, but the original Bophades seemed to help keep the models on track, so I was thinking that + Gutenberg again might do well there in helping both it's brains and creativity.
Either way, thanks again for your work done!
Hey, thanks for the feedback! I'll probably try tuning Mistral Small, but I'd like to do a little more testing on Nemo to find out which datasets might work best first.
I'm also working on a new dataset that I would like to try before renting hardware to tune Mistral Small.
Sure, sure. Makes sense to start small and see what sticks before moving upward.
If you want any help with dataset management, like ideas on new ones, or even basic cleanup, then I don't mind lending a hand for a bit. I tried to get my own hands dirty by doing actual training at first, looking at multiple interesting datasets and such, but never could figure out how to configure it all. So now I just have a stack on my drive just sitting there, lol.
Hi @nbeerbower . Sorry for hijacking this thread. I've really enjoyed Lyra Gutenberg and Gutenberg v3 for writing. I've been trying out Unslop Nemo v2 by TheDrummer. Compared to Rocinante, its prose has less GPT-isms and is quite unlike any other I've tried. Would it possible to make a writing finetune (Gutenberg or a mixture of other datasets) of that? Thanks for your work
Sure, sure. Makes sense to start small and see what sticks before moving upward.
If you want any help with dataset management, like ideas on new ones, or even basic cleanup, then I don't mind lending a hand for a bit. I tried to get my own hands dirty by doing actual training at first, looking at multiple interesting datasets and such, but never could figure out how to configure it all. So now I just have a stack on my drive just sitting there, lol.
Sure, I'm always open to dataset suggestions!
Hi @nbeerbower . Sorry for hijacking this thread. I've really enjoyed Lyra Gutenberg and Gutenberg v3 for writing. I've been trying out Unslop Nemo v2 by TheDrummer. Compared to Rocinante, its prose has less GPT-isms and is quite unlike any other I've tried. Would it possible to make a writing finetune (Gutenberg or a mixture of other datasets) of that? Thanks for your work
Yeah I can give that a shot. I've never tried loading GGUF straight into Transformers but it seems like it's natively supported: https://huggingface.co/docs/transformers/main/en/gguf
Okay! I won't bog you with too much, but these 3 I think are the most interesting, while being maybe not as well known.
https://huggingface.co/datasets/TheSkullery/Aether-Lite-PurHyDe
First one here is just an amalgamation of different stuff, kinda like Bagel, but with more heavy filtering and a creative focus. When I was thinking of doing a dataset myself this was kinda my starting point. If you already have a strong core for training then it might not be as worthwhile.
https://huggingface.co/datasets/davanstrien/cosmochat
Second is more so for general intelligence. This was similar to an idea I had before coming across it, where instead of just spitting factual data, you set up scenarios or chats between the two that is more natural sounding and walk them through the learning process. Very lightweight at 200 entries.
https://huggingface.co/datasets/chtmp223/suri
Third here seems to make it so the bot understands the context of System prompts and user requests, aligning it more towards what the intended effect should be when asked. Supposedly can be used for ORPO as well.
I'll end off here with a user here who has a cool collection of some of the better datasets, but shortened them into bite-sized chunks. Pretty much all of them I was eyeing at some point, so it's more so a general rec like with Aether if you want to add some stuff on top.
Anyway, hope your future training goes well!