Clean HTML to Markdown for LLM & RAG Pipelines: Token Minimization & Semantic Table Extraction

Retrieval-Augmented Generation (RAG) and LLM agent pipelines ingest millions of web pages, internal wikis, customer support tickets, and API documentation daily.

However, feeding raw HTML directly into LLM prompts or vector embeddings is disastrous:

1. Severe Token Bloat: Raw HTML contains <script>, <style>, <div>, <svg>, inline CSS, and nested attributes (class="flex flex-col md:w-1/2 p-4 text-gray-800 dark:text-gray-100"). HTML boilerplate routinely consumes 70% to 85% of total token budgets.
2. Degraded Retrieval Accuracy: Embedding models (like text-embedding-3-large or bge-m3) struggle to capture semantic relevance when poisoned by noisy tag hierarchies.
3. Broken Table Semantics: Unstructured text conversion destroys table row-and-column alignments, causing LLMs to hallucinate data correlations.

Converting sanitized HTML to clean, GitHub-Flavored Markdown (GFM) solves all three issues.

This guide presents the complete architecture for high-throughput HTML-to-Markdown processing in Node.js and Python.

---

1. The RAG Ingestion Pipeline: Before and After

[Raw Scraped HTML] (15,000 Tokens)
        |
        v
[1. DOM Sanitization & Boilerplate Stripping]
    - Drop <script>, <style>, <nav>, <footer>, <svg>, <noscript>
    - Extract semantic core (<article>, <main>, or Readability root)
        |
        v
[2. AST / Markdown Conversion]
    - Headings (#, ##, ###)
    - Code blocks with syntax highlighting language
    - Clean GitHub-Flavored Markdown tables
        |
        v
[3. Chunking & Token Budgeting]
        |
        v
[Clean GFM Output] (2,400 Tokens - 84% Reduction!)

---

2. Benchmark: Raw HTML vs Plain Text vs Markdown

Format	Token Count (Est.)	Semantic Structure	Table Integrity	Embedding Quality
Raw HTML	~14,200 tokens	Overloaded with DOM noise	Intact but noisy	Poor (distracted by class names)
Plain Text (innerText)	~1,800 tokens	Destroyed (all headings flattened)	Destroyed (cells merged into text blob)	Moderate (lacks hierarchy)
Clean Markdown (GFM)	~2,350 tokens	Preserved (#, ##, lists, code)	Preserved (pipe tables | A | B |)	Highest

---

3. High-Performance Implementation in Node.js / TypeScript

In Node.js, combine @mozilla/readability (for semantic content extraction) and turndown with the GFM plugin:

import { JSDOM } from 'jsdom';
import { Readability } from '@mozilla/readability';
import TurndownService from 'turndown';
import { gfm } from 'turndown-plugin-gfm';

export function htmlToCleanMarkdown(rawHtml: string): string {
  const dom = new JSDOM(rawHtml);
  const doc = dom.window.document;

  // 1. Remove non-content elements before parsing
  const elementsToRemove = doc.querySelectorAll(
    'script, style, noscript, svg, iframe, nav, footer, header, form, [aria-hidden="true"]'
  );
  elementsToRemove.forEach((el) => el.remove());

  // 2. Extract main article via Readability
  const reader = new Readability(doc);
  const article = reader.parse();
  const contentHtml = article?.content || doc.body.innerHTML;

  // 3. Configure Turndown with GFM (Tables, Strikethrough, Task lists)
  const turndown = new TurndownService({
    headingStyle: 'atx',
    hr: '---',
    bulletListMarker: '-',
    codeBlockStyle: 'fenced',
    emDelimiter: '_',
  });

  turndown.use(gfm);

  // Custom rule: Preserve code language tags
  turndown.addRule('fencedCodeBlock', {
    filter: (node) => node.nodeName === 'PRE' && !!node.querySelector('code'),
    replacement: (_content, node) => {
      const codeEl = (node as HTMLElement).querySelector('code');
      const lang = codeEl?.className?.match(/language-(\w+)/)?.[1] || '';
      return `\n\`\`\`${lang}\n${codeEl?.textContent?.trim() || ''}\n\`\`\`\n`;
    },
  });

  // 4. Convert and normalize excessive whitespace
  const markdown = turndown.turndown(contentHtml);
  return markdown.replace(/\n{3,}/g, '\n\n').trim();
}

---

4. High-Performance Implementation in Python (with markdownify & BeautifulSoup)

import re
from bs4 import BeautifulSoup
from markdownify import markdownify as md

def clean_html_to_markdown(raw_html: str) -> str:
    soup = BeautifulSoup(raw_html, 'html.parser')

    # 1. Strip useless tags
    for tag in soup(['script', 'style', 'nav', 'footer', 'aside', 'noscript', 'svg', 'form']):
        tag.decompose()

    # 2. Prefer <main> or <article> if available
    main_content = soup.find('main') or soup.find('article') or soup.body or soup

    # 3. Convert to markdown with table support
    markdown = md(
        str(main_content),
        heading_style="ATX",
        bullets="-",
        code_language="",
        strip=['img'] # Strip heavy images for pure text RAG
    )

    # 4. Clean extra newlines
    clean_md = re.sub(r'\n{3,}', '\n\n', markdown).strip()
    return clean_md

---

5. Preserving Complex Tables for LLM Reasoning

When LLMs answer questions based on tabular data (e.g., pricing sheets, specifications), standard text extraction collapses rows into unreadable strings.

Markdown tables maintain both column headers and row associations:

| Model | Context Window | Input Cost / 1M | Output Cost / 1M |
|---|---|---|---|
| GPT-4o | 128k | $2.50 | $10.00 |
| Claude 3.5 Sonnet | 200k | $3.00 | $15.00 |
| Gemini 1.5 Pro | 2000k | $1.25 | $5.00 |

When chunking Markdown for vector databases (like Pinecone, Qdrant, or Chroma):

• Never split inside a Markdown table. Ensure chunking logic treats the entire table as a single atomic unit.
• If a table exceeds max chunk size, replicate the table header row at the start of each sliced chunk.

---

6. Real-World RAG Chunking Pattern in TypeScript

export interface MarkdownChunk {
  content: string;
  headingHierarchy: string[];
}

export function chunkMarkdownByHeadings(markdown: string, maxTokens = 500): MarkdownChunk[] {
  const lines = markdown.split('\n');
  const chunks: MarkdownChunk[] = [];
  let currentHeadings: string[] = [];
  let currentBuffer: string[] = [];

  for (const line of lines) {
    const headingMatch = line.match(/^(#{1,6})\s+(.*)$/);
    if (headingMatch) {
      const level = headingMatch[1].length;
      const title = headingMatch[2];

      if (currentBuffer.length > 0) {
        chunks.push({
          content: currentBuffer.join('\n').trim(),
          headingHierarchy: [...currentHeadings],
        });
        currentBuffer = [];
      }

      currentHeadings = currentHeadings.slice(0, level - 1);
      currentHeadings[level - 1] = title;
    }

    currentBuffer.push(line);
  }

  if (currentBuffer.length > 0) {
    chunks.push({
      content: currentBuffer.join('\n').trim(),
      headingHierarchy: currentHeadings,
    });
  }

  return chunks;
}

Use the HTML to Markdown Converter to clean and preview extracted content, and check token savings with the AI Token Counter.