RAG文本切分LV3：轻松定制Markdown切分原创

恰似惊鸿

发布于 2024-9-18 14:55

浏览

0收藏

上篇文章我们介绍了借助LLM和OCR将文档转换成markdown的方法：颠覆传统OCR轻松搞定复杂PDF的工具。本篇文章将介绍如何对markdown进行有效切分。

之前介绍了文本切分五个层级，本文方法是第三个层次：

Level 1: Character Splitting - 简单的字符长度切分

Level 2: Recursive Character Text Splitting - 通过分隔符切分，然后递归合并

Level 3: Document Specific Splitting - 针对不同文档格式切分 (PDF, Python, Markdown)

Level 4: Semantic Splitting - 语义切分

Level 5: Agentic Splitting-使用代理实现自动切分

基本概念和环境

分块通常旨在将具有共同上下文的文本放在一起。考虑到这一点，我们可能希望特别尊重文档本身的结构。例如，markdown 文件按标题组织。在特定标题组中创建块是一种直观的想法。为了解决这一挑战，我们可以使用MarkdownHeaderTextSplitter。这将按指定的一组标题拆分 markdown 文件。

本文用到的安装包如下：

pip install langchain-text-splitters

切分实现

我们可以指定要拆分的标题headers_to_split_on，切分之后内容按标题分组：

markdown_document = "# Foo\n\n    ## Bar\n\nHi this is Jim\n\nHi this is Joe\n\n ### Boo \n\n Hi this is Lance \n\n ## Baz\n\n Hi this is Molly"

headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
    ("###", "Header 3"),
]

markdown_splitter = MarkdownHeaderTextSplitter(
  headers_to_split_on)
md_header_splits = markdown_splitter.split_text(
  markdown_document)
print(md_header_splits)

结果如下：

[Document(page_content='Hi this is Jim  \nHi this is Joe', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}),
Document(page_content='Hi this is Lance', metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}),
Document(page_content='Hi this is Molly', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'})]

默认情况下，MarkdownHeaderTextSplitter从输出块的内容中剥离被分割的标头。可以通过设置strip_headers = False来禁用此功能。

markdown_splitter = MarkdownHeaderTextSplitter(
headers_to_split_on,
strip_headers=False)
md_header_splits = markdown_splitter.split_text(
markdown_document)
print(md_header_splits)

可以看到，标题添加到内容中了

[Document(page_content='# Foo  \n## Bar  \nHi this is Jim  \nHi this is Joe', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}),
Document(page_content='### Boo  \nHi this is Lance', metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}),
Document(page_content='## Baz  \nHi this is Molly', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'})]

如何将 Markdown 行返回为单独的文档

默认情况下，MarkdownHeaderTextSplitter根据headers_to_split_on中指定的标题聚合行。我们可以通过指定return_each_line来禁用此功能，使得一行就是一条内容：

markdown_splitter = MarkdownHeaderTextSplitter(
headers_to_split_on,
return_each_line=True,
)
md_header_splits = markdown_splitter.split_text(markdown_document)
print(md_header_splits)

[Document(page_content='Hi this is Jim', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}),
Document(page_content='Hi this is Joe', metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}),
Document(page_content='Hi this is Lance', metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}),
Document(page_content='Hi this is Molly', metadata={'Header 1': 'Foo', 'Header 2': 'Baz'})]

如何限制块大小：

然后，我们可以在每个 markdown 组中应用任何我们想要的文本分割器，例如RecursiveCharacterTextSplitter，它允许进一步控制块大小。


markdown_document = "# Intro \n\n    ## History \n\n Markdown[9] is a lightweight markup language for creating formatted text using a plain-text editor. John Gruber created Markdown in 2004 as a markup language that is appealing to human readers in its source code form.[9] \n\n Markdown is widely used in blogging, instant messaging, online forums, collaborative software, documentation pages, and readme files. \n\n ## Rise and divergence \n\n As Markdown popularity grew rapidly, many Markdown implementations appeared, driven mostly by the need for \n\n additional features such as tables, footnotes, definition lists,[note 1] and Markdown inside HTML blocks. \n\n #### Standardization \n\n From 2012, a group of people, including Jeff Atwood and John MacFarlane, launched what Atwood characterised as a standardisation effort. \n\n ## Implementations \n\n Implementations of Markdown are available for over a dozen programming languages."

headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
]

# MD splits
markdown_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on, strip_headers=False
)
md_header_splits = markdown_splitter.split_text(markdown_document)

# Char-level splits
from langchain_text_splitters import RecursiveCharacterTextSplitter

chunk_size = 250
chunk_overlap = 30
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size, chunk_overlap=chunk_overlap
)

# Split
splits = text_splitter.split_documents(md_header_splits)
splits

本文转载自公众号哎呀AIYA

原文链接：https://mp.weixin.qq.com/s/58OJQoi-xuxdFhU02Q6uZg

标签

RAG

Markdown