Is this apply late chunking?
#1
by
thatpham2k
- opened
When working with LLM, we sometimes encounter context-sensitive chunking (aka late chunking) to ensure our model understands the context.
More detail: https://github.com/jina-ai/late-chunking/blob/main/README.md
I have tried late chunking with this embedding model, but it's seem not work out. I put this colab notebook here: https://colab.research.google.com/drive/1JYas5VJWModbJfGZak0XDN8KTDQyR3g6?usp=sharing. If you're free, please check that. The result is bad when compare with traditional chunking (not apply late chunking)