Skip to content

Parse long posts using LLM

Idea is to parse long posts using LLM: Simply pass it into OLLAMA with prompt to create a note from the post of maximum length of chunk size and then create embedding from this note.

This method should improve quality of embeddings for long post without need to chunk them into separate parts. Disadvantage is that it will increase time of embedding post because additional step will be added to the process.