​

Data Extraction

Delegate web scraping logic to an LLM, execute safely on AVM nodes, and obtain structured CSV/JSON without local risk.

​

Use Cases

​

Web2: Shopify Product APIs

Extract product data, pricing, and inventory information from e-commerce platforms.

​

Web3: CoinGecko, Twitter, Dune

Scrape cryptocurrency data, social sentiment, and blockchain analytics for comprehensive market analysis.

​

Scenario: Bulk Scraping

Extract account balances pages from a DeFi dashboard in parallel.

​

Implementation: Two-Stage Extraction

1. Fetch HTML
   Retrieve page content locally.
2. Parse with LLM
   Prompt the LLM to extract table data via BeautifulSoup.
3. Run in AVM
   Execute parsing code with the runPython tool.
4. Aggregate Results
   Combine CSV outputs for all URLs.

​

Example (TypeScript)

import { runPythonTool } from "@avm-ai/avm-vercel-ai";
import fetch from "node-fetch";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";

const tool = runPythonTool();

async function scrape(url: string) {
  const html = await fetch(url).then(r => r.text());
  const prompt = `
Write execute(input) that uses BeautifulSoup to parse input["data"] HTML 
and outputs CSV.`;
  const { text: code } = await generateText({
    model: openai("gpt-4o"),
    prompt,
    tools: { runPython: tool },
  });
  const result = await tool.exec({ code, input: { data: html } });
  return result.output.stdout;
}

​

Next Steps

• Parallelize jobs via MCP concurrency.
• Store results on IPFS.
• Add retry and throttling mechanisms.