Build Your First Tabstack App

Getting started

In this tutorial, you'll build a command-line tool that converts any documentation website into an AI-friendly AGENTS.md file. This file helps AI assistants understand and work with the documented API or tool.

What You’ll Build

A Node.js application that:

Takes a documentation URL as input
Extracts all documentation pages from the site
Reads and processes the content
Generates a comprehensive AGENTS.md file summarizing the documentation

What You’ll Learn

How to use the Tabstack API to convert web pages to markdown
How to extract and process multiple documentation pages
How to use AI-powered transformations to generate structured documentation

Prerequisites

Node.js 20+ installed
A Tabstack API key (get one here)
Basic command-line knowledge
20 minutes of your time

Setup Your Project

1. Create Your Project Directory

mkdir doc-parser
cd doc-parser
npm init -y

2. Install Dependencies

npm install axios dotenv

axios: For making HTTP requests to the Tabstack API
dotenv: For managing your API key securely

3. Set Up Your API Key

Create a .env file in your project root:

TABSTACK_API_KEY=your_api_key_here

Replace your_api_key_here with your actual API key from the Tabstack Console.

:::warning Never commit your .env file to version control! Add it to .gitignore:

echo ".env" >> .gitignore

:::

Understanding the Application Flow

Here’s how our application will work:

┌─────────────────┐
│  Documentation  │
│   Website URL   │
└────────┬────────┘
         │
         ▼
┌─────────────────────────┐
│  1. Fetch Main Page     │  ← /v1/extract/markdown endpoint
│     (Convert to MD)     │
└────────┬────────────────┘
         │
         ▼
┌─────────────────────────┐
│  2. Extract Doc Links   │  ← Parse markdown
│     from Content        │
└────────┬────────────────┘
         │
         ▼
┌─────────────────────────┐
│  3. Fetch All Doc       │  ← /v1/extract/markdown endpoint
│     Pages (Convert)     │     (multiple calls)
└────────┬────────────────┘
         │
         ▼
┌─────────────────────────┐
│  4. Generate AGENTS.md  │  ← /v1/generate/json endpoint
│     with AI             │
└────────┬────────────────┘
         │
         ▼
┌─────────────────────────┐
│   AGENTS.md File        │
│   Ready for AI Use!     │
└─────────────────────────┘

Building the Application

Step 1: Set Up the Basic Structure

Create index.js:

// Import required libraries
require("dotenv").config();
const axios = require("axios");
const fs = require("fs").promises;

// Configuration
const TABSTACK_API_BASE = "https://api.tabstack.ai";
const API_KEY = process.env.TABSTACK_API_KEY;

// Check if API key is set
if (!API_KEY) {
  console.error(
    "❌ Error: TABSTACK_API_KEY not found in environment variables"
  );
  console.error("Please create a .env file with your API key");
  process.exit(1);
}

// Main function
async function main() {
  // Get URL from command line arguments
  const url = process.argv[2];

  if (!url) {
    console.error("❌ Error: Please provide a documentation URL");
    console.error("Usage: node index.js <documentation-url>");
    process.exit(1);
  }

  console.log("🚀 Starting documentation parser...");
  console.log(`📄 Target URL: ${url}\n`);

  try {
    // We'll add our implementation here
    console.log("✅ Done!");
  } catch (error) {
    console.error("❌ Error:", error.message);
    process.exit(1);
  }
}

// Run the application
main();

Step 2: Fetch the Main Documentation Page

Add this helper function before the main() function:

/**
 * Fetch a URL and convert it to markdown using Tabstack API
 * @param {string} url - The URL to fetch
 * @returns {Promise<{url: string, content: string}>} The markdown content
 */
async function fetchAsMarkdown(url) {
  console.log(`📥 Fetching: ${url}`);

  try {
    const response = await axios.post(
      `${TABSTACK_API_BASE}/v1/extract/markdown`,
      { url },
      {
        headers: {
          Authorization: `Bearer ${API_KEY}`,
          "Content-Type": "application/json",
        },
      }
    );

    return response.data;
  } catch (error) {
    if (error.response) {
      throw new Error(
        `API Error: ${error.response.status} - ${error.response.data.error || "Unknown error"}`
      );
    }
    throw error;
  }
}

Now update the main() function to fetch the main page:

async function main() {
  const url = process.argv[2];

  if (!url) {
    console.error("❌ Error: Please provide a documentation URL");
    console.error("Usage: node index.js <documentation-url>");
    process.exit(1);
  }

  console.log("🚀 Starting documentation parser...");
  console.log(`📄 Target URL: ${url}\n`);

  try {
    // Step 1: Fetch the main documentation page
    console.log("Step 1: Fetching main documentation page...");
    const mainPage = await fetchAsMarkdown(url);
    console.log(`✅ Fetched ${mainPage.content.length} characters\n`);

    console.log("✅ Done!");
  } catch (error) {
    console.error("❌ Error:", error.message);
    process.exit(1);
  }
}

Step 3: Extract Documentation Links

Add this function to extract links from markdown content:

/**
 * Extract documentation links from markdown content
 * @param {string} markdown - The markdown content
 * @param {string} baseUrl - The base URL for resolving relative links
 * @returns {string[]} Array of unique documentation URLs
 */
function extractDocLinks(markdown, baseUrl) {
  console.log("🔍 Extracting documentation links...");

  // Regular expression to match markdown links: [text](url)
  const linkRegex = /\[([^\]]+)\]\(([^)]+)\)/g;
  const links = new Set();

  let match;
  while ((match = linkRegex.exec(markdown)) !== null) {
    let link = match[2];

    // Skip anchors, mailto, and external protocol links
    if (
      link.startsWith("#") ||
      link.startsWith("mailto:") ||
      link.startsWith("tel:")
    ) {
      continue;
    }

    // Convert relative URLs to absolute
    try {
      const absoluteUrl = new URL(link, baseUrl).href;

      // Only include links from the same domain
      const baseDomain = new URL(baseUrl).hostname;
      const linkDomain = new URL(absoluteUrl).hostname;

      if (linkDomain === baseDomain) {
        links.add(absoluteUrl);
      }
    } catch (e) {
      // Skip invalid URLs
      continue;
    }
  }

  const uniqueLinks = Array.from(links);
  console.log(`✅ Found ${uniqueLinks.length} documentation links\n`);

  return uniqueLinks;
}

Update main() to extract links:

async function main() {
  const url = process.argv[2];

  if (!url) {
    console.error("❌ Error: Please provide a documentation URL");
    console.error("Usage: node index.js <documentation-url>");
    process.exit(1);
  }

  console.log("🚀 Starting documentation parser...");
  console.log(`📄 Target URL: ${url}\n`);

  try {
    // Step 1: Fetch the main documentation page
    console.log("Step 1: Fetching main documentation page...");
    const mainPage = await fetchAsMarkdown(url);
    console.log(`✅ Fetched ${mainPage.content.length} characters\n`);

    // Step 2: Extract documentation links
    console.log("Step 2: Extracting documentation links...");
    const docLinks = extractDocLinks(mainPage.content, url);

    // Limit to first 10 pages to avoid rate limits (remove this for production)
    const linksToFetch = docLinks.slice(0, 10);
    console.log(`📚 Will fetch ${linksToFetch.length} documentation pages\n`);

    console.log("✅ Done!");
  } catch (error) {
    console.error("❌ Error:", error.message);
    process.exit(1);
  }
}

Step 4: Fetch All Documentation Pages

Add a function to fetch multiple pages with a delay to respect rate limits:

/**
 * Fetch multiple URLs as markdown with rate limiting
 * @param {string[]} urls - Array of URLs to fetch
 * @returns {Promise<Array<{url: string, content: string}>>} Array of markdown content
 */
async function fetchMultipleAsMarkdown(urls) {
  console.log("📥 Fetching documentation pages...");

  const results = [];

  for (let i = 0; i < urls.length; i++) {
    try {
      const result = await fetchAsMarkdown(urls[i]);
      results.push(result);
      console.log(`   ✅ ${i + 1}/${urls.length} complete`);

      // Add a small delay between requests to be respectful
      if (i < urls.length - 1) {
        await new Promise((resolve) => setTimeout(resolve, 500));
      }
    } catch (error) {
      console.log(`   ⚠️  Failed to fetch ${urls[i]}: ${error.message}`);
      // Continue with other URLs
    }
  }

  console.log(
    `✅ Successfully fetched ${results.length}/${urls.length} pages\n`
  );
  return results;
}

Update main() to fetch all documentation pages:

async function main() {
  const url = process.argv[2];

  if (!url) {
    console.error("❌ Error: Please provide a documentation URL");
    console.error("Usage: node index.js <documentation-url>");
    process.exit(1);
  }

  console.log("🚀 Starting documentation parser...");
  console.log(`📄 Target URL: ${url}\n`);

  try {
    // Step 1: Fetch the main documentation page
    console.log("Step 1: Fetching main documentation page...");
    const mainPage = await fetchAsMarkdown(url);
    console.log(`✅ Fetched ${mainPage.content.length} characters\n`);

    // Step 2: Extract documentation links
    console.log("Step 2: Extracting documentation links...");
    const docLinks = extractDocLinks(mainPage.content, url);

    // Limit to first 10 pages to avoid rate limits
    const linksToFetch = docLinks.slice(0, 10);
    console.log(`📚 Will fetch ${linksToFetch.length} documentation pages\n`);

    // Step 3: Fetch all documentation pages
    console.log("Step 3: Fetching all documentation pages...");
    const allPages = await fetchMultipleAsMarkdown([url, ...linksToFetch]);

    console.log("✅ Done!");
  } catch (error) {
    console.error("❌ Error:", error.message);
    process.exit(1);
  }
}

Step 5: Generate AGENTS.md with AI

Now for the magic! We’ll use the /transform endpoint to analyze all the documentation and generate a structured AGENTS.md file:

/**
 * Generate AGENTS.md content using AI transformation
 * @param {string} url - The base documentation URL
 * @param {Array<{url: string, content: string}>} pages - All fetched pages
 * @returns {Promise<string>} The generated AGENTS.md content
 */
async function generateAgentsMd(url, pages) {
  console.log("🤖 Generating AGENTS.md with AI...");

  // Combine all markdown content
  const combinedContent = pages
    .map((page) => {
      return `# Source: ${page.url}\n\n${page.content}\n\n---\n\n`;
    })
    .join("");

  // Create a temporary combined markdown file URL
  // For the transform endpoint, we'll use the original URL
  // and pass instructions for comprehensive analysis

  const instructions = `
Analyze this documentation comprehensively and create an AGENTS.md file that helps AI assistants understand and work with this API/tool.

The AGENTS.md should include:

1. **Project Overview**: Brief description of what this project/API does
2. **Key Concepts**: Important concepts, terminology, and architecture
3. **API Endpoints**: Summary of available endpoints (if applicable) with methods and purposes
4. **Authentication**: How to authenticate (if applicable)
5. **Common Use Cases**: Typical scenarios and workflows
6. **Code Examples**: Key usage patterns with code snippets from the docs
7. **Important Notes**: Rate limits, best practices, gotchas, limitations

Format the output as clear, well-structured markdown that an AI agent can easily parse and understand.
Focus on actionable information that helps an AI assistant answer questions and help users work with this tool.
  `.trim();

  // Define the schema for the output
  const schema = {
    type: "object",
    properties: {
      content: {
        type: "string",
        description: "The complete AGENTS.md file content in markdown format",
      },
    },
    required: ["content"],
    additionalProperties: false,
  };

  try {
    const response = await axios.post(
      `${TABSTACK_API_BASE}/v1/generate/json`,
      {
        url: url,
        instructions: instructions,
        json_schema: schema,
      },
      {
        headers: {
          Authorization: `Bearer ${API_KEY}`,
          "Content-Type": "application/json",
        },
      }
    );

    console.log("✅ AGENTS.md generated successfully\n");
    return response.data.content;
  } catch (error) {
    if (error.response) {
      throw new Error(
        `API Error: ${error.response.status} - ${error.response.data.error || "Unknown error"}`
      );
    }
    throw error;
  }
}

Step 6: Save the Generated File

Add a function to save the AGENTS.md file:

/**
 * Save content to a file
 * @param {string} filename - The filename to save to
 * @param {string} content - The content to save
 */
async function saveToFile(filename, content) {
  console.log(`💾 Saving to ${filename}...`);
  await fs.writeFile(filename, content, "utf8");
  console.log(`✅ Saved successfully\n`);
}

Step 7: Complete the Main Function

Now let’s tie it all together in the main() function:

async function main() {
  const url = process.argv[2];

  if (!url) {
    console.error("❌ Error: Please provide a documentation URL");
    console.error("Usage: node index.js <documentation-url>");
    process.exit(1);
  }

  console.log("🚀 Starting documentation parser...");
  console.log(`📄 Target URL: ${url}\n`);

  try {
    // Step 1: Fetch the main documentation page
    console.log("Step 1: Fetching main documentation page...");
    const mainPage = await fetchAsMarkdown(url);
    console.log(`✅ Fetched ${mainPage.content.length} characters\n`);

    // Step 2: Extract documentation links
    console.log("Step 2: Extracting documentation links...");
    const docLinks = extractDocLinks(mainPage.content, url);

    // Limit to first 10 pages to avoid rate limits
    const linksToFetch = docLinks.slice(0, 10);
    console.log(`📚 Will fetch ${linksToFetch.length} documentation pages\n`);

    // Step 3: Fetch all documentation pages
    console.log("Step 3: Fetching all documentation pages...");
    const allPages = await fetchMultipleAsMarkdown([url, ...linksToFetch]);

    // Step 4: Generate AGENTS.md
    console.log("Step 4: Generating AGENTS.md...");
    const agentsMd = await generateAgentsMd(url, allPages);

    // Step 5: Save to file
    await saveToFile("AGENTS.md", agentsMd);

    console.log("✅ Done! Your AGENTS.md file is ready.");
    console.log(`📝 Generated from ${allPages.length} documentation pages`);
  } catch (error) {
    console.error("❌ Error:", error.message);
    process.exit(1);
  }
}

Running Your Application

Now you can run your documentation parser!

node index.js https://docs.example.com

Example Output

🚀 Starting documentation parser...
📄 Target URL: https://docs.stripe.com/api

Step 1: Fetching main documentation page...
📥 Fetching: https://docs.stripe.com/api
✅ Fetched 45231 characters

Step 2: Extracting documentation links...
🔍 Extracting documentation links...
✅ Found 87 documentation links

📚 Will fetch 10 documentation pages

Step 3: Fetching all documentation pages...
📥 Fetching documentation pages...
   ✅ 1/10 complete
   ✅ 2/10 complete
   ...
✅ Successfully fetched 10/10 pages

Step 4: Generating AGENTS.md...
🤖 Generating AGENTS.md with AI...
✅ AGENTS.md generated successfully

💾 Saving to AGENTS.md...
✅ Saved successfully

✅ Done! Your AGENTS.md file is ready.
📝 Generated from 10 documentation pages

Example Generated AGENTS.md

Your generated file might look like this:

# Stripe API - AI Agent Documentation

## Project Overview

Stripe is a payment processing platform that provides APIs for accepting payments,
managing subscriptions, and handling financial transactions...

## Key Concepts

- **Payment Intents**: Represents the lifecycle of a customer payment
- **Customers**: Represents your customers in Stripe
- **Subscriptions**: Recurring payment schedules...

## API Endpoints

### Payments

- `POST /v1/payment_intents` - Create a payment intent
- `GET /v1/payment_intents/:id` - Retrieve payment details
- `POST /v1/payment_intents/:id/confirm` - Confirm payment

### Customers

- `POST /v1/customers` - Create a new customer
- `GET /v1/customers/:id` - Retrieve customer details...

[... more content ...]

Testing Your Application

Try it with different documentation sites:

# Stripe API docs
node index.js https://docs.stripe.com/api

# Tabstack API docs (meta!)
node index.js https://docs.tabstack.ai

# GitHub REST API docs
node index.js https://docs.github.com/en/rest

Complete Code

Here’s the complete index.js file for reference:

// Import required libraries
require("dotenv").config();
const axios = require("axios");
const fs = require("fs").promises;

// Configuration
const TABSTACK_API_BASE = "https://api.tabstack.ai";
const API_KEY = process.env.TABSTACK_API_KEY;

// Check if API key is set
if (!API_KEY) {
  console.error(
    "❌ Error: TABSTACK_API_KEY not found in environment variables"
  );
  console.error("Please create a .env file with your API key");
  process.exit(1);
}

/**
 * Fetch a URL and convert it to markdown using Tabstack API
 */
async function fetchAsMarkdown(url) {
  console.log(`📥 Fetching: ${url}`);

  try {
    const response = await axios.post(
      `${TABSTACK_API_BASE}/v1/extract/markdown`,
      { url },
      {
        headers: {
          Authorization: `Bearer ${API_KEY}`,
          "Content-Type": "application/json",
        },
      }
    );

    return response.data;
  } catch (error) {
    if (error.response) {
      throw new Error(
        `API Error: ${error.response.status} - ${error.response.data.error || "Unknown error"}`
      );
    }
    throw error;
  }
}

/**
 * Extract documentation links from markdown content
 */
function extractDocLinks(markdown, baseUrl) {
  console.log("🔍 Extracting documentation links...");

  const linkRegex = /\[([^\]]+)\]\(([^)]+)\)/g;
  const links = new Set();

  let match;
  while ((match = linkRegex.exec(markdown)) !== null) {
    let link = match[2];

    if (
      link.startsWith("#") ||
      link.startsWith("mailto:") ||
      link.startsWith("tel:")
    ) {
      continue;
    }

    try {
      const absoluteUrl = new URL(link, baseUrl).href;
      const baseDomain = new URL(baseUrl).hostname;
      const linkDomain = new URL(absoluteUrl).hostname;

      if (linkDomain === baseDomain) {
        links.add(absoluteUrl);
      }
    } catch (e) {
      continue;
    }
  }

  const uniqueLinks = Array.from(links);
  console.log(`✅ Found ${uniqueLinks.length} documentation links\n`);

  return uniqueLinks;
}

/**
 * Fetch multiple URLs as markdown with rate limiting
 */
async function fetchMultipleAsMarkdown(urls) {
  console.log("📥 Fetching documentation pages...");

  const results = [];

  for (let i = 0; i < urls.length; i++) {
    try {
      const result = await fetchAsMarkdown(urls[i]);
      results.push(result);
      console.log(`   ✅ ${i + 1}/${urls.length} complete`);

      if (i < urls.length - 1) {
        await new Promise((resolve) => setTimeout(resolve, 500));
      }
    } catch (error) {
      console.log(`   ⚠️  Failed to fetch ${urls[i]}: ${error.message}`);
    }
  }

  console.log(
    `✅ Successfully fetched ${results.length}/${urls.length} pages\n`
  );
  return results;
}

/**
 * Generate AGENTS.md content using AI transformation
 */
async function generateAgentsMd(url, pages) {
  console.log("🤖 Generating AGENTS.md with AI...");

  const instructions = `
Analyze this documentation comprehensively and create an AGENTS.md file that helps AI assistants understand and work with this API/tool.

The AGENTS.md should include:

1. **Project Overview**: Brief description of what this project/API does
2. **Key Concepts**: Important concepts, terminology, and architecture
3. **API Endpoints**: Summary of available endpoints (if applicable) with methods and purposes
4. **Authentication**: How to authenticate (if applicable)
5. **Common Use Cases**: Typical scenarios and workflows
6. **Code Examples**: Key usage patterns with code snippets from the docs
7. **Important Notes**: Rate limits, best practices, gotchas, limitations

Format the output as clear, well-structured markdown that an AI agent can easily parse and understand.
Focus on actionable information that helps an AI assistant answer questions and help users work with this tool.
  `.trim();

  const schema = {
    type: "object",
    properties: {
      content: {
        type: "string",
        description: "The complete AGENTS.md file content in markdown format",
      },
    },
    required: ["content"],
    additionalProperties: false,
  };

  try {
    const response = await axios.post(
      `${TABSTACK_API_BASE}/v1/generate/json`,
      {
        url: url,
        instructions: instructions,
        json_schema: schema,
      },
      {
        headers: {
          Authorization: `Bearer ${API_KEY}`,
          "Content-Type": "application/json",
        },
      }
    );

    console.log("✅ AGENTS.md generated successfully\n");
    return response.data.content;
  } catch (error) {
    if (error.response) {
      throw new Error(
        `API Error: ${error.response.status} - ${error.response.data.error || "Unknown error"}`
      );
    }
    throw error;
  }
}

/**
 * Save content to a file
 */
async function saveToFile(filename, content) {
  console.log(`💾 Saving to ${filename}...`);
  await fs.writeFile(filename, content, "utf8");
  console.log(`✅ Saved successfully\n`);
}

/**
 * Main function
 */
async function main() {
  const url = process.argv[2];

  if (!url) {
    console.error("❌ Error: Please provide a documentation URL");
    console.error("Usage: node index.js <documentation-url>");
    process.exit(1);
  }

  console.log("🚀 Starting documentation parser...");
  console.log(`📄 Target URL: ${url}\n`);

  try {
    // Step 1: Fetch the main documentation page
    console.log("Step 1: Fetching main documentation page...");
    const mainPage = await fetchAsMarkdown(url);
    console.log(`✅ Fetched ${mainPage.content.length} characters\n`);

    // Step 2: Extract documentation links
    console.log("Step 2: Extracting documentation links...");
    const docLinks = extractDocLinks(mainPage.content, url);

    // Limit to first 10 pages to avoid rate limits
    const linksToFetch = docLinks.slice(0, 10);
    console.log(`📚 Will fetch ${linksToFetch.length} documentation pages\n`);

    // Step 3: Fetch all documentation pages
    console.log("Step 3: Fetching all documentation pages...");
    const allPages = await fetchMultipleAsMarkdown([url, ...linksToFetch]);

    // Step 4: Generate AGENTS.md
    console.log("Step 4: Generating AGENTS.md...");
    const agentsMd = await generateAgentsMd(url, allPages);

    // Step 5: Save to file
    await saveToFile("AGENTS.md", agentsMd);

    console.log("✅ Done! Your AGENTS.md file is ready.");
    console.log(`📝 Generated from ${allPages.length} documentation pages`);
  } catch (error) {
    console.error("❌ Error:", error.message);
    process.exit(1);
  }
}

// Run the application
main();

Learn More

Ready to explore more?

API Reference - Detailed documentation of all endpoints
Quickstart Guide - Get your API key and make your first call
Examples - More real-world examples

Need Help?

Check out the full API documentation
Support: [email protected]