Text model not being loaded with Flash Attention 2
#27
by
starzmustdie
- opened
No description provided.
yes you are correct
@starzmustdie
we left that as an improvement when integrating into hf transformers, but I just opened a public issue to track this https://github.com/huggingface/transformers/issues/30394
Hey @VictorSanh
After some time debugging, this was the reason why I was getting OOM when trying to fine-tune the model.
I opened a PR which patches this (https://github.com/huggingface/transformers/pull/30395).
I would appreciate feedback on necessary changes.
should be fixed with https://github.com/huggingface/transformers/pull/30507
closing this
VictorSanh
changed pull request status to
closed