之前介绍了文本切分五个层级，本文方法是第三个层次：
-
Level 1: Character Splitting – 简单的字符长度切分
-
Level 2: Recursive Character Text Splitting – 通过分隔符切分，然后递归合并
-
Level 3: Document Specific Splitting – 针对不同文档格式切分 (PDF, Python, Markdown)
-
Level 4: Semantic Splitting – 语义切分
-
Level 5: Agentic Splitting-使用代理实现自动切分
基本概念和环境
分块通常旨在将具有共同上下文的文本放在一起。考虑到这一点，我们可能希望特别尊重文档本身的结构。例如，markdown 文件按标题组织。在特定标题组中创建块是一种直观的想法。为了解决这一挑战，我们可以使用MarkdownHeaderTextSplitter。这将按指定的一组标题拆分 markdown 文件。
本文用到的安装包如下：
pip install langchain-text-splitters
切分实现
我们可以指定要拆分的标题headers_to_split_on，切分之后内容按标题分组 ：
markdown_document = "# Foonn## BarnnHi this is JimnnHi this is Joenn ### Boo nn Hi this is Lance nn ## Baznn Hi this is Molly"headers_to_split_on = [("#", "Header 1"),("##", "Header 2"),("###", "Header 3"),]markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on)md_header_splits = markdown_splitter.split_text(markdown_document)print(md_header_splits)
结果如下：
[Document(page_content='Hi this is JimnHi this is Joe', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}), Document(page_content='Hi this is Lance', metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}), Document(page_content='Hi this is Molly', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'})]
markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on, strip_headers=False)md_header_splits = markdown_splitter.split_text(markdown_document)print(md_header_splits)
[Document(page_content='# Foon## BarnHi this is JimnHi this is Joe', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}), Document(page_content='### BoonHi this is Lance', metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}), Document(page_content='## BaznHi this is Molly', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'})]
如何将 Markdown 行返回为单独的文档
默认情况下，MarkdownHeaderTextSplitter根据headers_to_split_on中指定的标题聚合行。我们可以通过指定return_each_line来禁用此功能，使得一行就是一条内容：
markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on,return_each_line=True,)md_header_splits = markdown_splitter.split_text(markdown_document)print(md_header_splits)
[Document(page_content='Hi this is Jim', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}), Document(page_content='Hi this is Joe', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}), Document(page_content='Hi this is Lance', metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}), Document(page_content='Hi this is Molly', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'})]
如何限制块大小：
然后，我们可以在每个 markdown 组中应用任何我们想要的文本分割器，例如RecursiveCharacterTextSplitter，它允许进一步控制块大小。
markdown_document = "# Intro nn## History nn Markdown[9] is a lightweight markup language for creating formatted text using a plain-text editor. John Gruber created Markdown in 2004 as a markup language that is appealing to human readers in its source code form.[9] nn Markdown is widely used in blogging, instant messaging, online forums, collaborative software, documentation pages, and readme files. nn ## Rise and divergence nn As Markdown popularity grew rapidly, many Markdown implementations appeared, driven mostly by the need for nn additional features such as tables, footnotes, definition lists,[note 1] and Markdown inside HTML blocks. nn #### Standardization nn From 2012, a group of people, including Jeff Atwood and John MacFarlane, launched what Atwood characterised as a standardisation effort. nn ## Implementations nn Implementations of Markdown are available for over a dozen programming languages."headers_to_split_on = [("#", "Header 1"),("##", "Header 2"),]# MD splitsmarkdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on, strip_headers=False)md_header_splits = markdown_splitter.split_text(markdown_document)# Char-level splitsfrom langchain_text_splitters import RecursiveCharacterTextSplitterchunk_size = 250chunk_overlap = 30text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)# Splitsplits = text_splitter.split_documents(md_header_splits)splits
颠覆传统OCR轻松搞定复杂PDF的工具
