Imported from TheGraphicKing/skott-canvas (skills/competitor-scraper/SKILL.md). Install upstream with npx skills add TheGraphicKing/skott-canvas --skill competitor-scraper. Copyright stays with the author.

Overview

The competitor-scraper skill fetches public web content from six competitor source types and returns structured, deduplicated artifact payloads. It operates as the foundational data-collection layer of the competitor intelligence pipeline. Every artifact produced by this skill carries full provenance metadata so downstream classifiers, theme extractors, and gap-analysis agents can trace each signal back to its origin without re-fetching the source.

The skill is designed around precise structured extraction, not raw HTML forwarding. It parses each source type with purpose-built extraction logic and returns only the fields downstream agents need — saving token budget, avoiding hallucination risk in downstream agents, and enabling robust deduplication.

When to Use

When the competitor-scheduler dispatches a scan job for a specific competitor
When a one-off competitive intelligence request requires fresh data from a specific source type
When the gap-analysis-agent needs a spot-check refresh of a competitor's pricing or careers page
When the weekly-digest-generator needs a manual pull of recent press mentions before digest assembly

How to Execute

Step 1: Job Validation

Validate the incoming job payload: confirm that competitor ID, source URL list, run ID, and last-scan timestamp are all present. If any required field is missing, return an error and do not proceed. Load the competitor's entry from the competitor-intel knowledge file to confirm the source URL list is current — the knowledge file is the source of truth, not the job payload.

Step 2: Blog Scraping

Fetch the competitor's blog or news index. Parse all posts with a publication date after the last-scan timestamp. For each post extract:

title (string)
url (string)
published_at (ISO 8601 date)
author (string, if available)
word_count (integer)
body_text (string, full text)
topic_tags (array of strings)
product_claims (array: any named features, capabilities, or integrations mentioned)
customer_references (array: any named customers, logos, or case study references)
data_citations (array: any third-party data, analyst reports, or statistics cited with source)

Limit to 20 most recent posts unless the scheduler has set a high-activity override.

Step 3: LinkedIn Post Scraping

Retrieve recent public LinkedIn company posts. For each post extract:

post_text (string)
published_at (ISO 8601 date)
post_type (enum: organic, product-announcement, event-promo, hiring, customer-story, thought-leadership)
engagement_signals (object: likes, comments, reposts — include only if visible without authentication)
external_links (array: any URLs linked from the post)
lyzr_mention (boolean: does the post name Lyzr or a Lyzr-adjacent term?)

Step 4: Pricing Page Scraping

Fetch the competitor's pricing page. Extract:

pricing_model (enum: per-seat, consumption, flat-rate, enterprise-only, freemium)
tier_names (array of strings)
tier_descriptions (array: one string per tier)
feature_gates (array: features listed as tier-specific)
deployment_options (array: cloud, on-prem, hybrid — only if explicitly stated)
cta_language (string: the primary call-to-action text, e.g., "Start Free Trial" or "Contact Sales")
minimum_contract_signals (string: any language implying deal size floor, e.g., "for teams of 50+")
diff_from_previous (boolean: has this page changed since the last scrape?)

Step 5: Careers Page Scraping

Fetch the competitor's open roles. Aggregate by department. Extract:

total_open_roles (integer)
roles_by_department (object: department name → count)
ai_ml_engineering_count (integer)
sales_count (integer)
vertical_specific_roles (array: role titles that name a vertical — BFSI, healthcare, manufacturing)
net_new_roles_vs_prior (integer: positive = growth, negative = shrinkage)
notable_senior_roles (array: VP, C-suite, or Director-level roles newly posted)

Step 6: G2 Review Scraping

Retrieve the most recent G2 reviews from the competitor's product listing page. For each review extract:

star_rating (integer 1-5)
reviewed_at (ISO 8601 date)
reviewer_title (string)
reviewer_company_size (enum: 1-50, 51-200, 201-1000, 1001-5000, 5000+)
pros_text (string)
cons_text (string)
features_praised (array: named product features mentioned positively)
features_criticized (array: named product features mentioned negatively)
high_signal_flag (boolean: true if review mentions enterprise deployment, pricing, implementation complexity, or support response time)

Do not extract reviewer names — reviewer title and company size are sufficient for signal analysis.

Step 7: Press Mention Scraping

Search configured press sources and Google News for the competitor name within the scan window. For each mention extract:

headline (string)
publication (string)
published_at (ISO 8601 date)
url (string)
summary (string: 2-3 sentence summary of the article's relevance)
mention_type (enum: product-launch, partnership, funding, analyst-coverage, award, executive-change, negative-coverage, other)
lyzr_mentioned (boolean: does the article also mention Lyzr or a direct Lyzr competitor alongside this competitor?)

Step 8: Deduplication and Output Assembly

For every extracted artifact, compute a fingerprint: SHA-256 hash of (source_type + source_url + published_at + title). Check the artifact store for the fingerprint. If the fingerprint exists, discard the artifact and increment the duplicate count. Only write net-new artifacts to the output payload.

Each output artifact must include: competitor_id, source_type, artifact_id (UUID), run_id, fingerprint, scraped_at (ISO 8601), and the structured fields for that source type.

Inputs

job_payload (object, required): Competitor ID, source URL list, run ID, last-scan timestamp
artifact_store_reference (string, required): Reference to the artifact store for deduplication checks
high_activity_override (boolean, optional): If true, lift the 20-post limit for blog scraping

Outputs

artifacts (array): Array of structured artifact objects, one per net-new item discovered
duplicate_count (integer): Number of artifacts discarded as duplicates
source_errors (array): List of source URLs that returned errors, with status codes
job_completion_status (enum: complete, partial, failed)

Quality Criteria

Every artifact must include all required metadata fields — an artifact with a missing run_id or fingerprint is invalid and must be discarded
No fabricated or inferred fields — if a field cannot be extracted directly from the page content, it must be null, not estimated
Deduplication must run before output assembly — duplicate artifacts must never appear in the output array
Source errors must be reported accurately — a 429 rate-limit error and a 404 not-found error require different handling by the scheduler and must be reported with the correct status code

competitor-scraper

About