# Token Types and Interfaces

Token types representing parsed markdown elements and their structure.

## Token Type Hierarchy

```typescript { .api }
/**
 * Union of all standard token types
 * These are the built-in token types recognized by marked
 */
type MarkedToken =
  | Tokens.Blockquote
  | Tokens.Br
  | Tokens.Checkbox
  | Tokens.Code
  | Tokens.Codespan
  | Tokens.Def
  | Tokens.Del
  | Tokens.Em
  | Tokens.Escape
  | Tokens.Heading
  | Tokens.Hr
  | Tokens.HTML
  | Tokens.Image
  | Tokens.Link
  | Tokens.List
  | Tokens.ListItem
  | Tokens.Paragraph
  | Tokens.Space
  | Tokens.Strong
  | Tokens.Table
  | Tokens.Tag
  | Tokens.Text;

/**
 * Union of all token types including custom/generic tokens
 * Use this type when working with tokens from extensions
 */
type Token = MarkedToken | Tokens.Generic;

/**
 * Array of tokens with link references
 * Returned by Lexer.lex()
 */
type TokensList = Token[] & {
  /**
   * Map of link reference IDs to their href and title
   * Keys are normalized to lowercase
   */
  links: Links;
};

/**
 * Map of link reference IDs to their href and title
 * Used for resolving [text][ref] syntax
 */
type Links = Record<string, Pick<Tokens.Link | Tokens.Image, 'href' | 'title'>>;
```

## Block-Level Tokens

### Space Token

```typescript { .api }
namespace Tokens {
  interface Space {
    type: 'space';
    /**
     * Raw whitespace/newline content
     */
    raw: string;
  }
}
```

Represents whitespace and newlines between block elements. Usually not rendered (returns empty string).

**Example Token**:
```javascript
{
  type: 'space',
  raw: '\n\n'
}
```

### Code Token

```typescript { .api }
namespace Tokens {
  interface Code {
    type: 'code';
    /**
     * Raw source including fences or indentation
     */
    raw: string;
    /**
     * Programming language identifier (from fence info string)
     * Empty string if not specified
     */
    lang?: string;
    /**
     * Code content (without fences or indentation)
     */
    text: string;
    /**
     * Whether this is an indented code block (vs fenced)
     * Only present for indented code blocks
     */
    codeBlockStyle?: 'indented';
    /**
     * Whether special characters are already escaped
     * If true, don't escape again during rendering
     */
    escaped?: boolean;
  }
}
```

Represents fenced code blocks (`` ``` ``) and indented code blocks (4+ spaces).

**Example Token**:
```javascript
{
  type: 'code',
  raw: '```javascript\nconsole.log("hi");\n```',
  lang: 'javascript',
  text: 'console.log("hi");',
  escaped: false
}
```

**Fenced vs Indented**:
- Fenced: Can specify language, more explicit
- Indented: Has `codeBlockStyle: 'indented'`

### Heading Token

```typescript { .api }
namespace Tokens {
  interface Heading {
    type: 'heading';
    /**
     * Raw heading source
     */
    raw: string;
    /**
     * Heading level (1-6)
     * 1 = h1, 2 = h2, etc.
     */
    depth: number;
    /**
     * Heading text (unparsed, may contain markdown)
     */
    text: string;
    /**
     * Inline tokens within heading
     * Parse these to get formatted heading text
     */
    tokens: Token[];
  }
}
```

Represents ATX (`# Heading`) and Setext (underlined) headings.

**Example Token**:
```javascript
{
  type: 'heading',
  raw: '# **Bold** Heading',
  depth: 1,
  text: '**Bold** Heading',
  tokens: [
    { type: 'strong', text: 'Bold', tokens: [...], raw: '**Bold**' },
    { type: 'text', text: ' Heading', raw: ' Heading' }
  ]
}
```

### Horizontal Rule Token

```typescript { .api }
namespace Tokens {
  interface Hr {
    type: 'hr';
    /**
     * Raw HR source (---, ***, ___)
     */
    raw: string;
  }
}
```

Represents horizontal rules. Requires at least 3 dashes, asterisks, or underscores.

**Example Token**:
```javascript
{
  type: 'hr',
  raw: '---'
}
```

### Blockquote Token

```typescript { .api }
namespace Tokens {
  interface Blockquote {
    type: 'blockquote';
    /**
     * Raw blockquote source (with > markers)
     */
    raw: string;
    /**
     * Unparsed blockquote content (without > markers)
     */
    text: string;
    /**
     * Block tokens within blockquote
     * Parse these to get formatted blockquote content
     */
    tokens: Token[];
  }
}
```

Represents blockquotes (`> quote`). Can contain nested markdown.

**Example Token**:
```javascript
{
  type: 'blockquote',
  raw: '> This is a quote\n> with multiple lines',
  text: 'This is a quote\nwith multiple lines',
  tokens: [
    { type: 'paragraph', text: 'This is a quote\nwith multiple lines', tokens: [...], raw: '...' }
  ]
}
```

### List Token

```typescript { .api }
namespace Tokens {
  interface List {
    type: 'list';
    /**
     * Raw list source
     */
    raw: string;
    /**
     * Whether this is an ordered (numbered) list
     * true = ol, false = ul
     */
    ordered: boolean;
    /**
     * Starting number for ordered lists
     * Empty string for unordered lists
     */
    start: number | '';
    /**
     * Whether list items are separated by blank lines
     * Affects paragraph wrapping in items
     */
    loose: boolean;
    /**
     * Array of list item tokens
     */
    items: ListItem[];
  }
}
```

Represents ordered (numbered) and unordered (bulleted) lists.

**Example Token**:
```javascript
{
  type: 'list',
  raw: '1. First\n2. Second',
  ordered: true,
  start: 1,
  loose: false,
  items: [
    { type: 'list_item', raw: '1. First', text: 'First', tokens: [...], task: false, loose: false },
    { type: 'list_item', raw: '2. Second', text: 'Second', tokens: [...], task: false, loose: false }
  ]
}
```

**Ordered vs Unordered**:
- Ordered: `ordered: true`, has `start` number
- Unordered: `ordered: false`, `start: ''`

**Loose vs Tight**:
- Loose: Items wrapped in `<p>` tags
- Tight: Items not wrapped

### List Item Token

```typescript { .api }
namespace Tokens {
  interface ListItem {
    type: 'list_item';
    /**
     * Raw list item source
     */
    raw: string;
    /**
     * Whether this is a task list item (GFM)
     */
    task: boolean;
    /**
     * Whether task list checkbox is checked (if task is true)
     * Only present when task is true
     */
    checked?: boolean;
    /**
     * Whether item is separated from others by blank lines
     */
    loose: boolean;
    /**
     * Unparsed item content
     */
    text: string;
    /**
     * Block tokens within list item
     */
    tokens: Token[];
  }
}
```

Represents individual list items, including task list items (GFM feature).

**Example Token (regular)**:
```javascript
{
  type: 'list_item',
  raw: '- Item text',
  task: false,
  loose: false,
  text: 'Item text',
  tokens: [...]
}
```

**Example Token (task list)**:
```javascript
{
  type: 'list_item',
  raw: '- [x] Completed task',
  task: true,
  checked: true,
  loose: false,
  text: 'Completed task',
  tokens: [...]
}
```

### Table Token (GFM)

```typescript { .api }
namespace Tokens {
  interface Table {
    type: 'table';
    /**
     * Raw table source
     */
    raw: string;
    /**
     * Column alignments (null = no alignment)
     */
    align: Array<'center' | 'left' | 'right' | null>;
    /**
     * Table header cells
     */
    header: TableCell[];
    /**
     * Table body rows (each row is an array of cells)
     */
    rows: TableCell[][];
  }

  interface TableCell {
    /**
     * Unparsed cell text
     */
    text: string;
    /**
     * Inline tokens within cell
     */
    tokens: Token[];
    /**
     * Whether this is a header cell
     */
    header: boolean;
    /**
     * Cell alignment (inherited from column)
     */
    align: 'center' | 'left' | 'right' | null;
  }

  interface TableRow<P = string> {
    /**
     * Rendered row content (used internally by renderer)
     * Contains HTML of all cells concatenated
     */
    text: P;
  }
}
```

Represents GitHub Flavored Markdown tables.

**Example Token**:
```javascript
{
  type: 'table',
  raw: '| A | B |\n|---|---|\n| 1 | 2 |',
  align: [null, null],
  header: [
    { text: 'A', tokens: [...], header: true, align: null },
    { text: 'B', tokens: [...], header: true, align: null }
  ],
  rows: [
    [
      { text: '1', tokens: [...], header: false, align: null },
      { text: '2', tokens: [...], header: false, align: null }
    ]
  ]
}
```

**Alignment Options**:
- `null`: No alignment
- `'left'`: Left-aligned (`:---`)
- `'right'`: Right-aligned (`---:`)
- `'center'`: Center-aligned (`:---:`)

### Paragraph Token

```typescript { .api }
namespace Tokens {
  interface Paragraph {
    type: 'paragraph';
    /**
     * Raw paragraph source
     */
    raw: string;
    /**
     * Whether this paragraph is in a special context
     * (e.g., inside HTML block where normal paragraph rules don't apply)
     */
    pre?: boolean;
    /**
     * Unparsed paragraph text
     */
    text: string;
    /**
     * Inline tokens within paragraph
     */
    tokens: Token[];
  }
}
```

Represents paragraph blocks.

**Example Token**:
```javascript
{
  type: 'paragraph',
  raw: 'This is a paragraph with **bold** text.',
  text: 'This is a paragraph with **bold** text.',
  tokens: [
    { type: 'text', text: 'This is a paragraph with ', raw: '...' },
    { type: 'strong', text: 'bold', tokens: [...], raw: '**bold**' },
    { type: 'text', text: ' text.', raw: ' text.' }
  ]
}
```

### HTML Block Token

```typescript { .api }
namespace Tokens {
  interface HTML {
    type: 'html';
    /**
     * Raw HTML source
     */
    raw: string;
    /**
     * Whether this is a <pre> tag
     * Affects whitespace handling
     */
    pre: boolean;
    /**
     * HTML content
     */
    text: string;
    /**
     * Whether this is a block-level HTML element
     */
    block: boolean;
  }
}
```

Represents block-level HTML. Passed through without modification (not sanitized).

**Example Token**:
```javascript
{
  type: 'html',
  raw: '<div class="custom">\nContent\n</div>',
  pre: false,
  text: '<div class="custom">\nContent\n</div>',
  block: true
}
```

**Security Warning**: HTML tokens are not sanitized. Use DOMPurify or similar on output.

### Link Definition Token

```typescript { .api }
namespace Tokens {
  interface Def {
    type: 'def';
    /**
     * Raw definition source
     */
    raw: string;
    /**
     * Reference label/tag (normalized to lowercase)
     */
    tag: string;
    /**
     * Link URL
     */
    href: string;
    /**
     * Link title (empty string if not provided)
     */
    title: string;
  }
}
```

Represents link reference definitions (`[label]: url "title"`). These are not rendered but stored in `TokensList.links`.

**Example Token**:
```javascript
{
  type: 'def',
  raw: '[ref]: https://example.com "Example"',
  tag: 'ref',
  href: 'https://example.com',
  title: 'Example'
}
```

**Note**: Def tokens don't appear in rendered output. They populate the `links` object for reference resolution.

## Inline-Level Tokens

### Escape Token

```typescript { .api }
namespace Tokens {
  interface Escape {
    type: 'escape';
    /**
     * Raw escape sequence (e.g., \*)
     */
    raw: string;
    /**
     * Escaped character (e.g., *)
     */
    text: string;
  }
}
```

Represents escaped characters (`\*`, `\_`, etc.). Used to prevent markdown interpretation.

**Example Token**:
```javascript
{
  type: 'escape',
  raw: '\\*',
  text: '*'
}
```

**Common Escapes**: `\*`, `\_`, `\[`, `\]`, `\(`, `\)`, `\#`, `\+`, `\-`, `\.`, `\!`

### HTML Tag Token

```typescript { .api }
namespace Tokens {
  interface Tag {
    type: 'html';
    /**
     * Raw HTML tag
     */
    raw: string;
    /**
     * Whether tag is inside a link
     * Affects parsing (links can't contain links)
     */
    inLink: boolean;
    /**
     * Whether tag is inside a raw HTML block
     */
    inRawBlock: boolean;
    /**
     * HTML tag content
     */
    text: string;
    /**
     * Whether this is a block-level tag
     */
    block: boolean;
  }
}
```

Represents inline HTML tags.

**Example Token**:
```javascript
{
  type: 'html',
  raw: '<span class="custom">',
  inLink: false,
  inRawBlock: false,
  text: '<span class="custom">',
  block: false
}
```

### Link Token

```typescript { .api }
namespace Tokens {
  interface Link {
    type: 'link';
    /**
     * Raw link source
     */
    raw: string;
    /**
     * Link URL
     */
    href: string;
    /**
     * Link title (hover text)
     * null if not provided
     */
    title?: string | null;
    /**
     * Link text (unparsed, may contain markdown)
     */
    text: string;
    /**
     * Inline tokens within link text
     */
    tokens: Token[];
  }
}
```

Represents links in various formats: `[text](url)`, `[text](url "title")`, `[text][ref]`, `<url>`.

**Example Token**:
```javascript
{
  type: 'link',
  raw: '[**bold** link](https://example.com "Title")',
  href: 'https://example.com',
  title: 'Title',
  text: '**bold** link',
  tokens: [
    { type: 'strong', text: 'bold', tokens: [...], raw: '**bold**' },
    { type: 'text', text: ' link', raw: ' link' }
  ]
}
```

**Link Formats**:
- Inline: `[text](url)`
- With title: `[text](url "title")`
- Reference: `[text][ref]`
- Shortcut reference: `[ref]`
- Autolink: `<url>`
- GFM autolink: Raw URL

### Image Token

```typescript { .api }
namespace Tokens {
  interface Image {
    type: 'image';
    /**
     * Raw image source
     */
    raw: string;
    /**
     * Image URL
     */
    href: string;
    /**
     * Image title attribute
     * null if not provided
     */
    title: string | null;
    /**
     * Alt text (unparsed, may contain markdown)
     */
    text: string;
    /**
     * Inline tokens within alt text
     */
    tokens: Token[];
  }
}
```

Represents images: `![alt](url)`, `![alt](url "title")`, `![alt][ref]`.

**Example Token**:
```javascript
{
  type: 'image',
  raw: '![Alt text](image.png "Title")',
  href: 'image.png',
  title: 'Title',
  text: 'Alt text',
  tokens: [
    { type: 'text', text: 'Alt text', raw: 'Alt text' }
  ]
}
```

**Accessibility**: Always include meaningful alt text in `text` property.

### Strong Token

```typescript { .api }
namespace Tokens {
  interface Strong {
    type: 'strong';
    /**
     * Raw bold syntax (** or __)
     */
    raw: string;
    /**
     * Unparsed bold text
     */
    text: string;
    /**
     * Inline tokens within bold text
     */
    tokens: Token[];
  }
}
```

Represents bold/strong text: `**bold**` or `__bold__`.

**Example Token**:
```javascript
{
  type: 'strong',
  raw: '**bold text**',
  text: 'bold text',
  tokens: [
    { type: 'text', text: 'bold text', raw: 'bold text' }
  ]
}
```

**Nesting**: Can contain other inline elements except other strong tags.

### Emphasis Token

```typescript { .api }
namespace Tokens {
  interface Em {
    type: 'em';
    /**
     * Raw italic syntax (* or _)
     */
    raw: string;
    /**
     * Unparsed italic text
     */
    text: string;
    /**
     * Inline tokens within italic text
     */
    tokens: Token[];
  }
}
```

Represents italic/emphasis text: `*italic*` or `_italic_`.

**Example Token**:
```javascript
{
  type: 'em',
  raw: '*italic text*',
  text: 'italic text',
  tokens: [
    { type: 'text', text: 'italic text', raw: 'italic text' }
  ]
}
```

**Bold + Italic**: `***text***` produces nested strong and em tokens.

### Codespan Token

```typescript { .api }
namespace Tokens {
  interface Codespan {
    type: 'codespan';
    /**
     * Raw code syntax (with backticks)
     */
    raw: string;
    /**
     * Code content (without backticks)
     */
    text: string;
  }
}
```

Represents inline code: `` `code` ``.

**Example Token**:
```javascript
{
  type: 'codespan',
  raw: '`const x = 1`',
  text: 'const x = 1'
}
```

**Multiple Backticks**: ``` `` code with ` backtick `` ``` also supported.

### Line Break Token

```typescript { .api }
namespace Tokens {
  interface Br {
    type: 'br';
    /**
     * Raw break source (two spaces + \n, or \n with breaks: true)
     */
    raw: string;
  }
}
```

Represents hard line breaks. Requires two spaces + newline, or just newline with `breaks: true`.

**Example Token**:
```javascript
{
  type: 'br',
  raw: '  \n'
}
```

**Break Types**:
- Standard: Two spaces + newline
- GFM breaks: Single newline (requires `breaks: true`)

### Strikethrough Token (GFM)

```typescript { .api }
namespace Tokens {
  interface Del {
    type: 'del';
    /**
     * Raw strikethrough syntax (~~)
     */
    raw: string;
    /**
     * Unparsed strikethrough text
     */
    text: string;
    /**
     * Inline tokens within strikethrough text
     */
    tokens: Token[];
  }
}
```

Represents strikethrough text: `~~deleted~~`. GFM feature.

**Example Token**:
```javascript
{
  type: 'del',
  raw: '~~deleted text~~',
  text: 'deleted text',
  tokens: [
    { type: 'text', text: 'deleted text', raw: 'deleted text' }
  ]
}
```

**Requires**: `gfm: true` option.

### Text Token

```typescript { .api }
namespace Tokens {
  interface Text {
    type: 'text';
    /**
     * Raw text source
     */
    raw: string;
    /**
     * Text content
     */
    text: string;
    /**
     * Optional nested tokens
     * Present when text contains inline elements that were parsed
     */
    tokens?: Token[];
    /**
     * Whether special characters are already escaped
     */
    escaped?: boolean;
  }
}
```

Represents plain text. Fallback for content that doesn't match other patterns.

**Example Token**:
```javascript
{
  type: 'text',
  raw: 'plain text',
  text: 'plain text'
}
```

**With Nested Tokens** (less common):
```javascript
{
  type: 'text',
  raw: 'text with emphasis',
  text: 'text with emphasis',
  tokens: [...]
}
```

### Checkbox Token (GFM)

```typescript { .api }
namespace Tokens {
  interface Checkbox {
    type: 'checkbox';
    /**
     * Raw checkbox syntax ([ ] or [x])
     */
    raw: string;
    /**
     * Whether checkbox is checked
     */
    checked: boolean;
  }
}
```

Represents task list checkboxes: `[ ]` or `[x]`. Part of list items. GFM feature.

**Example Token**:
```javascript
{
  type: 'checkbox',
  raw: '[x]',
  checked: true
}
```

**Usage Context**: Only appears in task list items.

## Generic Token

For custom extensions:

```typescript { .api }
namespace Tokens {
  interface Generic {
    /**
     * Any custom properties
     */
    [index: string]: any;
    /**
     * Token type identifier (required)
     * Must be unique to avoid conflicts
     */
    type: string;
    /**
     * Raw matched markdown text (required)
     * Must match exactly what was consumed from source
     */
    raw: string;
    /**
     * Optional child tokens
     * Used by walkTokens to traverse nested content
     */
    tokens?: Token[];
  }
}
```

Extensions create custom tokens that conform to this interface:

**Example Custom Token**:
```javascript
{
  type: 'emoji',
  raw: ':wave:',
  name: 'wave',
  // Any other custom properties
}
```

**Custom Token Guidelines**:
- Always include `type` and `raw`
- Make `type` unique
- Add `tokens` array if contains nested markdown
- Document custom properties

## Common Token Properties

All tokens have:

- **`type`**: String identifying the token type (required)
- **`raw`**: The original markdown text that was matched (required)

Many tokens have:

- **`text`**: Unparsed text content
- **`tokens`**: Nested tokens (for tokens that can contain other elements)

**Token Property Patterns**:
- Block tokens: Usually have `tokens` array
- Inline tokens: May have `tokens` for nested content
- Terminal tokens (like `hr`, `br`): Only have `type` and `raw`

## Token Usage Examples

### Accessing Token Properties

```javascript { .api }
import { marked } from "marked";

const tokens = marked.lexer('# Hello **world**');

console.log(tokens[0]);
// {
//   type: 'heading',
//   raw: '# Hello **world**',
//   depth: 1,
//   text: 'Hello **world**',
//   tokens: [
//     { type: 'text', raw: 'Hello ', text: 'Hello ' },
//     { type: 'strong', raw: '**world**', text: 'world', tokens: [...] }
//   ]
// }

// Access nested tokens
console.log(tokens[0].tokens[1].type); // 'strong'
```

### Modifying Tokens

```javascript { .api }
import { marked } from "marked";

marked.use({
  walkTokens(token) {
    // Add custom property to all headings
    if (token.type === 'heading') {
      token.id = token.text.toLowerCase().replace(/\s+/g, '-');
      token.level = `h${token.depth}`; // Custom property
    }
    
    // Modify links
    if (token.type === 'link') {
      // Add flag for external links
      if (token.href.startsWith('http')) {
        token.external = true;
      }
    }
  }
});

const html = marked.parse('# My Heading\n[Link](https://example.com)');
```

**Token Modification Guidelines**:
- Modify in `walkTokens` before rendering
- Don't change `type` or `raw` (breaks parsing)
- Add custom properties as needed
- Modifications affect rendering

### Creating Custom Tokens

```javascript { .api }
import { marked } from "marked";

marked.use({
  extensions: [{
    name: 'mention',
    level: 'inline',
    start(src) {
      return src.indexOf('@');
    },
    tokenizer(src) {
      const match = src.match(/^@(\w+)/);
      if (match) {
        return {
          type: 'mention',
          raw: match[0],
          username: match[1],
          // Custom properties
          href: `/users/${match[1]}`
        };
      }
    },
    renderer(token) {
      return `<a href="${token.href}" class="mention">@${token.username}</a>`;
    }
  }]
});

// Usage: Hello @john
// Creates mention token: { type: 'mention', raw: '@john', username: 'john', href: '/users/john' }
```

### Token Type Checking

```javascript { .api }
import { marked } from "marked";

const tokens = marked.lexer(markdown);

// Type checking
tokens.forEach(token => {
  switch (token.type) {
    case 'heading':
      console.log(`Heading (h${token.depth}): ${token.text}`);
      break;
    case 'paragraph':
      console.log(`Paragraph: ${token.text}`);
      break;
    case 'list':
      console.log(`List (${token.ordered ? 'ordered' : 'unordered'}): ${token.items.length} items`);
      break;
    case 'code':
      console.log(`Code block (${token.lang || 'no language'})`);
      break;
    default:
      console.log(`Token: ${token.type}`);
  }
});
```

### Token Tree Traversal

```javascript { .api }
import { marked } from "marked";

function traverseTokens(tokens, callback, depth = 0) {
  tokens.forEach(token => {
    callback(token, depth);
    
    // Recursively traverse nested tokens
    if (token.tokens) {
      traverseTokens(token.tokens, callback, depth + 1);
    }
    
    // Handle list items
    if (token.type === 'list' && token.items) {
      token.items.forEach(item => {
        if (item.tokens) {
          traverseTokens(item.tokens, callback, depth + 1);
        }
      });
    }
    
    // Handle tables
    if (token.type === 'table') {
      token.header.forEach(cell => {
        if (cell.tokens) {
          traverseTokens(cell.tokens, callback, depth + 1);
        }
      });
      token.rows.forEach(row => {
        row.forEach(cell => {
          if (cell.tokens) {
            traverseTokens(cell.tokens, callback, depth + 1);
          }
        });
      });
    }
  });
}

// Usage
const tokens = marked.lexer(markdown);
traverseTokens(tokens, (token, depth) => {
  console.log('  '.repeat(depth) + token.type);
});
```

### Token Statistics

```javascript { .api }
import { marked } from "marked";

function analyzeTokens(markdown) {
  const tokens = marked.lexer(markdown);
  const stats = {
    headings: { h1: 0, h2: 0, h3: 0, h4: 0, h5: 0, h6: 0 },
    paragraphs: 0,
    codeBlocks: 0,
    links: 0,
    images: 0,
    lists: 0,
    blockquotes: 0,
    tables: 0
  };
  
  marked.walkTokens(tokens, token => {
    switch (token.type) {
      case 'heading':
        stats.headings[`h${token.depth}`]++;
        break;
      case 'paragraph':
        stats.paragraphs++;
        break;
      case 'code':
        stats.codeBlocks++;
        break;
      case 'link':
        stats.links++;
        break;
      case 'image':
        stats.images++;
        break;
      case 'list':
        stats.lists++;
        break;
      case 'blockquote':
        stats.blockquotes++;
        break;
      case 'table':
        stats.tables++;
        break;
    }
  });
  
  return stats;
}

// Usage
const stats = analyzeTokens(markdown);
console.log('Document statistics:', stats);
```

### Token Serialization

```javascript { .api }
import { marked } from "marked";

// Convert tokens to JSON
const tokens = marked.lexer(markdown);
const json = JSON.stringify(tokens, null, 2);
console.log(json);

// Parse back from JSON
const parsedTokens = JSON.parse(json);

// Restore links property (arrays lose custom properties in JSON)
parsedTokens.links = tokens.links;

// Can now parse
const html = marked.parser(parsedTokens);
```

**Serialization Notes**:
- Tokens are JSON-serializable
- Links object must be restored after parsing
- Useful for caching token trees
- Custom properties are preserved

### Token Validation

```javascript { .api }
import { marked } from "marked";

function validateTokens(tokens) {
  const errors = [];
  
  marked.walkTokens(tokens, token => {
    // Check required properties
    if (!token.type) {
      errors.push('Token missing type property');
    }
    if (!token.raw) {
      errors.push(`Token ${token.type} missing raw property`);
    }
    
    // Type-specific validation
    if (token.type === 'heading' && (token.depth < 1 || token.depth > 6)) {
      errors.push(`Invalid heading depth: ${token.depth}`);
    }
    if (token.type === 'link' && !token.href) {
      errors.push(`Link missing href: ${token.raw}`);
    }
    if (token.type === 'image' && !token.text) {
      errors.push(`Image missing alt text: ${token.href}`);
    }
  });
  
  return errors;
}

// Usage
const tokens = marked.lexer(markdown);
const errors = validateTokens(tokens);
if (errors.length > 0) {
  console.error('Token validation errors:', errors);
}
```
