About

Imported from bighardperson/computer-science-skills-collection (skills/crawl4ai-skill/SKILL.md). Install upstream with npx skills add bighardperson/computer-science-skills-collection --skill crawl4ai-skill. Copyright stays with the author (MIT-0).

Crawl4AI Skill - Web Crawler & Scraper

Name: crawl4ai-skill
Author: bighardperson

Web Crawling 网页爬虫 | Web Scraping 网页爬取 | LLM 优化输出

智能网页爬虫和爬取工具，支持搜索、全站爬取、动态页面抓取。Free web crawler and scraper with LLM-optimized Markdown output.

核心功能 | Core Features

🔍 Web Search 网页搜索 - DuckDuckGo search, 免 API key
🕷️ Web Crawling 网页爬虫 - Site crawler, spider, sitemap 识别
📝 Web Scraping 网页抓取 - Smart scraper, data extraction
📄 LLM-Optimized Output - Fit Markdown, 省 Token 80%
⚡ Dynamic Page Scraping - JavaScript 渲染页面爬取

快速开始 | Quick Start

安装 | Installation

pip install crawl4ai-skill

Web Search | 网页搜索

# Search the web with DuckDuckGo
crawl4ai-skill search "python web scraping"

Web Scraping | 单页爬取

# Scrape a single web page
crawl4ai-skill crawl https://example.com

Web Crawling | 全站爬虫

# Crawl entire website / spider
crawl4ai-skill crawl-site https://docs.python.org --max-pages 50

使用场景 | Use Cases

场景 1：Web Crawler for Documentation | 文档站爬虫

# Crawl documentation site with spider
crawl4ai-skill crawl-site https://docs.fastapi.com --max-pages 100

爬虫效果 | Crawler Output:

❌ 移除：导航栏、侧边栏、广告
✅ 保留：标题、正文、代码块
📊 Token：50,000 → 10,000（-80%）

场景 2：Search + Scrape | 搜索+爬取

# Search and scrape top results
crawl4ai-skill search-and-crawl "Vue 3 best practices" --crawl-top 3

场景 3：Dynamic Page Scraping | 动态页面抓取

JavaScript 渲染的页面爬取（雪球、知乎等）：

# Scrape JavaScript-heavy pages
crawl4ai-skill crawl https://xueqiu.com/S/BIDU --wait-until networkidle --delay 2

命令参考 | Commands

命令 Command	说明 Description
`search <query>`	Web search 网页搜索
`crawl <url>`	Web scraping 单页爬取
`crawl-site <url>`	Web crawling 全站爬虫
`search-and-crawl <query>`	Search + scrape 搜索并爬取

常用参数 | Common Options

# Web Search 搜索
--num-results 10          # Number of results

# Web Scraping 爬取
--format fit_markdown     # Output format
--output result.md        # Output file
--wait-until networkidle  # Wait strategy for dynamic pages
--delay 2                 # Additional wait time (seconds)
--wait-for ".selector"    # Wait for specific element

# Web Crawling 爬虫
--max-pages 100          # Max pages to crawl
--max-depth 3            # Max crawl depth

输出格式 | Output Formats

fit_markdown（推荐 Recommended）

智能提取，节省 80% Token。Smart extraction, save 80% tokens.

crawl4ai-skill crawl https://example.com --format fit_markdown

raw_markdown

保留完整结构。Preserve full structure.

crawl4ai-skill crawl https://example.com --format raw_markdown

为什么选择这个爬虫？| Why This Crawler?

✅ 免费爬虫 Free Crawler - 无需 API key，开箱即用
✅ 智能爬取 Smart Scraper - 自动去噪，提取核心内容
✅ 全站爬虫 Site Crawler - 支持 sitemap，递归爬取
✅ 动态爬取 Dynamic Scraping - JavaScript 渲染页面支持
✅ 搜索集成 Search Integration - DuckDuckGo 搜索内置

链接 | Links

📦 PyPI
💻 GitHub
🦞 ClawHub

Use it

Copy one of these into your project. Installing also returns the manifest and these snippets.

yaml

targets:
  - https://api.opensmartroute.ai/api/v1/registry/bighardperson-computer-science-skills-collection-crawl4ai-skill/manifest   # or paste the manifest below

Manifest

An Open Capability Manifest: the router reads it to know what this does, what it costs and when to pick it.

bighardperson-computer-science-skills-collection-crawl4ai-skill.ocm.jsonjson

{
  "ocm": "1",
  "id": "bighardperson-computer-science-skills-collection-crawl4ai-skill",
  "kind": "skill",
  "name": "crawl4ai-skill",
  "description": "Web crawling and scraping tool with LLM-optimized output. 网页爬虫爬取工具 | Web crawler, web scraper, spider. DuckDuckGo search, site crawling, dynamic page scraping. 智能搜索爬取 | Free, no API key required.",
  "publisher": "bighardperson",
  "version": "1.0.0",
  "capabilities": {
    "domains": [
      "coding"
    ],
    "tags": [
      "skill-md",
      "web-scraping",
      "web-crawling",
      "crawler",
      "scraper",
      "spider",
      "search",
      "markdown",
      "llm",
      "token-optimization"
    ],
    "languages": [
      "en"
    ]
  },
  "quality_prior": 0.6,
  "examples": [
    "Web crawling and scraping tool with LLM-optimized output. 网页爬虫爬取工具 | Web crawler, web scraper, spider. DuckDuckGo search, site crawling, dynamic page scraping. 智能搜索爬取 | Free, no API key required."
  ],
  "primary": false,
  "metadata": {
    "source": {
      "provider": "skills.sh",
      "repository": "https://github.com/bighardperson/computer-science-skills-collection",
      "path": "skills/crawl4ai-skill/SKILL.md",
      "ref": "HEAD",
      "url": "https://github.com/bighardperson/computer-science-skills-collection/blob/HEAD/skills/crawl4ai-skill/SKILL.md",
      "key": "bighardperson/computer-science-skills-collection/skills/crawl4ai-skill/SKILL.md"
    },
    "license": "MIT-0"
  },
  "instructions": "# Crawl4AI Skill - Web Crawler & Scraper\n\n**Web Crawling 网页爬虫 | Web Scraping 网页爬取 | LLM 优化输出**\n\n智能网页爬虫和爬取工具，支持搜索、全站爬取、动态页面抓取。Free web crawler and scraper with LLM-optimized Markdown output.\n\n## 核心功能 | Core Features\n\n- 🔍 **Web Search 网页搜索** - DuckDuckGo search, 免 API key\n- 🕷️ **Web Crawling 网页爬虫** - Site crawler, spider, sitemap 识别\n- 📝 **Web Scraping 网页抓取** - Smart scraper, data extraction\n- 📄 **LLM-Optimized Output** - Fit Markdown, 省 Token 80%\n- ⚡ **Dynamic Page Scraping** - JavaScript 渲染页面爬取\n\n---\n\n## 快速开始 | Quick Start\n\n### 安装 | Installation\n\n```bash\npip install crawl4ai-skill\n```\n\n### Web S",
  "cost": {
    "context_tokens": 774
  }
}

Fetch it by URL: GET /api/v1/registry/bighardperson-computer-science-skills-collection-crawl4ai-skill/manifest?version=1.0.0

Reviews

Star ratings from people who tried it. One review per account; edit yours any time.

No reviews yet. Install it, try it, and be the first to rate it.