Commit 93bf70b2 authored by Eric Frias's avatar Eric Frias
Browse files

Improve chunking long posts by ensuring we break the text on sentence

boundaries, and targeting a number of tokens we put in each chunk
instead of targeting a character count.
parent 59639bc0
Loading
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment