Jobs

Imported from nagmudit/Jobs (AGENTS.md). Install upstream with npx skills add nagmudit/Jobs. Copyright stays with the author.

Jobs — Agent Instructions

A personal job-search corpus tool. jobsearch/ pulls listings from Wellfound, RemoteOK, Himalayas, vickybytes and company ATS boards (Greenhouse, Ashby, Workable, Lever) into SQLite and serves a local filter/sort UI with apply links. wellfound-probe/ is the completed research that established how to read Wellfound at all — frozen evidence, not a live component.

There is no scorer, recommender, or auto-apply here, by design. The user filters; the tool fetches and displays. Assisted apply pre-fills an ATS form in a visible browser; the user reviews and submits, and the tool never does (ADR-017).

Source of truth

This repository is authoritative. Conversation history is not. If you learned something in chat that isn't written here, it does not survive this session — write it down or lose it.

When sources disagree, trust in this order:

1. Code, tests, schemas, config — what actually runs
2. Accepted ADRs in docs/architecture/decisions/
3. Active plans in docs/plans/active/
4. Engineering docs under docs/
5. wellfound-probe/REPORT.md — accurate as of 2026-09-02, describes a third-party site that can change under us
6. Everything else, including this file

Conflicts get investigated and recorded, never silently resolved in favour of whichever is easiest.

Before you start

1. Read docs/index.md and follow it to what your task touches — not everything.
2. Check docs/plans/active/ for work already underway in that area.
3. Read the implementation and its tests.
4. Run git status.
5. For anything substantial, write or update an execution plan first.

Stack

Python 3.11+ (developed on 3.14). httpx, beautifulsoup4, lxml, pyyaml, fastapi, uvicorn, pytest. SQLite (stdlib) with a WAL journal. No build step, no deploy, no container; CI is GitHub Actions only (see Commands). wellfound-probe/ additionally uses playwright for one probe module.

Layout

Commands

Run from jobsearch/ unless noted. Full list in docs/engineering/commands.md; by-task cheat sheet in docs/quick-reference.md.

A cold fetch across all seven sources is ~1,150 requests / 75-95 min. Run it in the background; re-runs inside cache_ttl_hours are cheap.

All verified 2026-09-06. There is no lint or typecheck configured — don't claim one ran. CI (.github/workflows/ci.yml) runs the suite on 3.11 and 3.14, both validators, and the suite again inside a network namespace with no route out (never an iptables -P OUTPUT DROP — that also severs the runner's own control channel and hangs the job). It also runs daily, because a dated fixture can go red with no commit to blame.

Conduct rules — these are not style preferences

This tool talks to a live third-party site that has no API and no agreement with us. Violating any of these can get the user's IP or account blocked, which breaks the tool for everyone using it.

• robots.txt is enforced in code, in src/fetch.py. /_jobs/ is disallowed. Re-verify before crawling any new path family; don't assume.
• A site owner's out-of-band permission is honoured only when declared in targets.yaml under robots_overrides, with who granted it and when. It flips one path prefix on one origin, never a host and never a pattern; robots.txt is still read, an unreadable one still refuses to crawl blind, and deleting the entry revokes it. Never bypass the check any other way. See ADR-013.
• 3–5 s between requests, concurrency 1. src/config.py rejects a delay_range floor below 3.0 s at load time. Do not remove that guard.
• No proxies, no CAPTCHA services, no stealth plugins, no TLS/fingerprint spoofing. If something only works by defeating a challenge, it does not go in this repo.
• The User-Agent is a rotating pool (user_agents in targets.yaml), drawn from per request, carrying no name, address or account link. This was a deliberate policy change on 2026-09-08 — the rule previously forbade it. See ADR-014, which records the trade-offs, including that it does not make the client look like those browsers (httpx's TLS fingerprint is unchanged) and that it removes the ability to ask a site owner for permission the way ADR-013 did. Empty the list to revert.
• Stop all traffic to a host on its first mitigation. cf-ray and cf-mitigated are logged for every response. A non-None mitigation or a 403/429/503 raises MitigationDetected, and that origin gets no further request of any kind for the rest of the run — enforced in Fetcher.get, which raises HostBlocked before robots, throttle, cache or transport. Other hosts carry on; the run exits 3 and the daily job still fails. Never retry through it, never add a backoff-and-continue path. Integrity errors (SchemaDrift etc.) still halt everything. See ADR-016.
• All network access goes through Fetcher.get. Adding a second path out to the network bypasses every rule above. There is one exemption, ADR-017: Assist's headed browser opens the one form the user clicked, on an exact host allowlist, and never clicks, submits, or touches a CAPTCHA. Its guards are tested in tests/test_assist_browser.py. Never weaken them, and never add a second exemption — src/cli.py sync takes a downloaded file path rather than fetching it, precisely so this stays true.
• Never build CAPTCHA or bot-check circumvention, for assist or anything else. A challenge is the human's to solve.
• Listings expire from the cache; a mitigation never does. Search pages, feeds and ATS boards pass max_age=fetcher.listing_ttl (cache_ttl_hours, default 6); detail pages pass none. A cached 403/429/503 or cf-mitigated is sticky forever — aging one out is a backoff-and-continue path on a timer. See ADR-011.
• robots.txt is cached per origin. One Fetcher may span hosts; each host is judged by its own rules. Never collapse that cache — doing so lets one site's robots.txt permit a fetch another site forbids. A host whose robots.txt cannot be read raises (RobotsUnreadable), is never crawled, and is blocked for the rest of the run like a mitigation. See ADR-007, ADR-016.

Conventions

The rules you cannot work without are below. The full list lives in docs/engineering/conventions.md — read it before changing anything under src/. It was split out on 2026-09-06 because it outgrew the line budget that keeps this file readable.

• Store raw, derive later. Apollo nodes go into job_raw.raw_json verbatim; filterable columns are derived from it and served by the jobs view (stored in job_derived, kept exact by triggers). Re-deriving is free, re-crawling is not. See ADR-002, ADR-015.
• The four assertions raise, never warn. SilentRoleFallback, PageWrap, YieldFloor, SchemaDrift guard failures that return HTTP 200 with plausible data. Downgrading one to a log line silently corrupts the corpus. See ADR-003.
• Nothing about roles or locations is hardcoded. They live in targets.yaml. A literal role slug in src/ is a bug.
• CORE_COLUMNS is positional. UNION ALL aligns by position, not name. assert_core_views checks it on every connect — never bypass it.
• IDs are namespaced "<source>:<native_id>". Mixing the upsert_job and add_provenance id forms silently orphans provenance. See ADR-008.
• Unparsed beats wrongly parsed. Return NULL for anything unrecognised, and always keep the raw string alongside.

Testing contract

• New P0/P1 behaviour → a quality/test-manifest.yaml entry in the same change.
• Bug fix → write the failing test first, watch it fail, then fix.
• Verify a test can fail. Break the behaviour, see red, restore it.
• Never assert on mocks of our own code. Fakes belong at the network boundary only.
• Tests must not hit the network. Fixtures live in tests/fixtures.py.
• State one of: tests added (what), existing tests cover it (which), or no test needed (why). An unstated decision is not acceptable.
• Never claim the suite passed without running it — paste the output.

Before you finish

• Tests actually run, not assumed.
• Docs updated for anything that changed behaviour, schema, config, or conduct rules.
• Active execution plan updated with what happened and what's left.
• Report validation you did not run, and why.

Never

• Commit secrets, .env values, or anything personal. The repo is public, and the user_agent in targets.yaml goes out in every request header and sits in public git history — keep it free of names, addresses and account links.
• Commit jobs.db or cache/ (both gitignored). They are large and regenerable.
• Claim tests passed without running them.
• Weaken an assertion or a conduct guard to make something pass.
• Extend wellfound-probe/ — it is a finished artifact. New work goes in jobsearch/.