mapj

Imported from Mario-pereyra/mapj (AGENTS.md). Install upstream with npx skills add Mario-pereyra/mapj. Copyright stays with the author.

AGENTS.md — Working on this repo as an AI agent

Hand-off doc for AI agents (Claude, Cursor, Continue, etc.) that take over work on this codebase. Read this first before making changes.

What this repo is

mapj is a Go CLI + MCP server for TOTVS ecosystem agents. Two binaries:

• mapj.exe — CLI for humans / scripts. Subcommands: search, fetch, tdn, confluence, protheus, mcp serve, auth, health, observability.
• mapj-mcp.exe — standalone MCP server over stdio JSON-RPC (MCP 2025-06-18), exposes 5 tools: search_totvs_docs, fetch_totvs_doc, list_totvs_filters, list_tdn_spaces, protheus_query. All with outputSchema + annotations.

All output is envelope-shaped {ok, command, result|error} and supports 4 output formats (auto, llm, toon, json). Exit codes follow a fixed contract (0/1/2/3/4) declared in internal/errors/codes.go.

Architecture in one screen

cmd/
├── mapj/main.go              CLI entrypoint (Cobra)
└── mapj-mcp/main.go          MCP server entrypoint

internal/
├── cli/                      Cobra command definitions
│   ├── root.go               registers all subcommands
│   ├── search.go             multi-source search
│   ├── fetch.go              webfetch router driver
│   ├── mcp.go                MCP serve command
│   ├── tdn.go, confluence.go, protheus.go, auth.go, health.go ...
├── output/                   Envelope + TOON/LLM/JSON formatters
├── errors/                   Exit code typed errors (ExitCoder)
└── logging/                  zap global logger + trace IDs

pkg/                          Library packages, one per source/feature
├── cst/                      TOTVS CST aggregator (BUSCA + BUSCA_FILTROS)
├── search/                   Orchestrator: fan-out, normalize, dedup, rank
├── forum/                    Discourse forum.totvs.io
├── wp/                       WordPress espacolegislacao.totvs.com
├── poui/                     PO UI static dataset
├── zendesk/                  Direct Zendesk Help Center (public + WithCookie)
├── youtube/                  yt-dlp subprocess + VTT→MD parser
├── brave/                    Brave Search API (opt-in via BRAVE_API_KEY)
├── fetch/                    Host-detection router + handlers
├── confluence/               (pre-existing) TDN/Confluence client + MD converter
├── protheus/                 (pre-existing) Protheus SQL Server queries
└── tds/                      (pre-existing) TOTVS Language Server bridge

mcp/                          MCP protocol implementation
├── server.go                 JSON-RPC 2.0 loop, write mutex, WaitGroup
├── handlers.go               initialize, tools/list, tools/call dispatch
├── schema.go                 tool descriptions + JSON Schemas
└── server_test.go            10 tests

testdata/                     Real responses captured from live endpoints
├── cst/        forum/        wp/        poui/        zendesk/

skills/                       SKILL.md files for AI agents
├── mapj/                     orchestrator
├── mapj-search/              v0.5.0
├── mapj-mcp/                 v0.5.0  (5 tools)
├── mapj-tdn-search/          pre-existing
├── mapj-confluence-export/   pre-existing
└── mapj-protheus-query/      pre-existing

docs/                         Extended docs (see docs/README.md for index)
├── confluence-export-guide.md
├── protheus-guide.md
├── websearch-benchmark-plan.md
├── zendesk-auth-playwright-plan.md   ← PLANNED, not implemented
├── prd-token-efficiency.md           ← v0.6 token efficiency PRD
├── tools-analysis-2026.md            ← Tool-by-tool audit vs SOTA 2026
├── benchmark-results/                ← v0.5-rerun + multilang data
└── known-issues/                     ← Fixed bug write-ups

Conventions you must follow

1. Envelopes & exit codes

Every CLI command emits exactly one JSON envelope to stdout. Errors use typed internal/errors constructors so exit codes are correct:

return &internalerrors.UsageError{Msg: "..."}        // exit 2
return &internalerrors.RetryableError{Msg: "..."}    // exit 4

Get a formatter via cli.GetFormatter() (honors --output/--json/--verbose). Wrap results with output.NewEnvelope(...) or output.NewErrorEnvelopeWithHint(...).

2. HTTP client pattern

Every adapter has its own client.go mirroring pkg/confluence/client.go: struct with BaseURL, UserAgent, httpClient, a private do() method with retry/backoff/jitter on 5xx and network errors. Inject traceId from internal/logging.GetTraceID() into every log line.

When adding a new source: copy the pattern from pkg/forum/client.go — that's the cleanest minimal example.

3. Logging

• All logs go to stderr via zap.L() (global, set up by internal/logging).
• Never write to stdout in library code — stdout is reserved for envelopes and the MCP JSON-RPC stream.
• Add zap.String("component", "<source>") to every log line so traces are grep-able.

4. Tests

• Use httptest.NewServer to mock HTTP endpoints; never call live URLs in tests (they break offline / in CI).
• Fixture pattern: capture a real response once into testdata/<source>/, commit the file, load it in tests with os.ReadFile.
• Tests must pass with go test ./... and go vet ./... clean.

5. Encoding

The CST endpoint returns latin-1 mislabeled as UTF-8. pkg/cst/client.go has fixEncodingIfMojibake() that detects and re-decodes. Any new adapter that talks to TOTVS endpoints should test for this — it's a common gotcha. Look for �� or \xEF\xBF\xBD in responses.

PowerShell on Windows shows �� for valid UTF-8 due to cp1252 console — that's not a real bug. Verify with python -c "..." reading the file.

6. Multi-source search additions

If adding a new source adapter (Layer 2):

1. Create pkg/<source>/client.go + types.go + client_test.go.
2. Add a normalize<Source>Hit function in pkg/search/normalize_others.go.
3. Wire it into the fan-out in pkg/search/engine.go (Search method).
4. Add Source<Source> constant in pkg/search/types.go.
5. Add weight in defaultSourceWeights and priority in defaultDedupPriority.
6. Extend parseSources in internal/cli/search.go.
7. Update skills/mapj-search/SKILL.md capability list.

7. Calling search_totvs_docs from an agent — sweet-spot strategy

Empirically validated (see docs/benchmark-results/summary.md, 48 runs × 6 variants × 8 queries on 2026-05-15):

There is no single best parameter set. The agent should pick parameters adaptively based on the query type. Two regimes:

A. Default regime (Protheus técnico específico, ~60% of queries) — códigos AdvPL, errores SEFAZ, integraciones, pontos de entrada:

{ "query": "<text>", "max_results": 10 }

• Hits 7 native adapters in parallel (~900ms median).
• Adding --include-web HURTS top-5: Exa con includeDomains trae falsos positivos similares (e.g. para MA440RNT trae MA410RPV).

B. Web-augmented regime (consultor funcional + legislação + UI + how-to) — activar include_web: true cuando la query coincide con alguna de:

{ "query": "<text>", "max_results": 15, "include_web": true }

• Para reforma tributária IBS CBS: regimen A → 1 hit, regimen B → 12 hits relevantes.
• Para po-page slot exemplo: regimen A → top-5 sin nada de po-ui, regimen B → top-5 con manuais + foro discussions.

✅ Fixeado en v0.5.0 (era "no-fix v0.4.1"): Con --include-web, los items que Exa devuelve apuntando a tdn.totvs.com ahora retienen la etiqueta nativa (tdn/central). El dedup da prioridad a la fuente nativa sobre web para el mismo URL. Ver docs/known-issues/dedup-web-overrides-native.md.

Defaults para evitar:

• max_results > 25: el dedup deja típicamente 9-17 finales, aumentar no aporta.
• freshness_days > 0: docs TOTVS de 2018-2020 siguen siendo válidos; filtrar baja recall fuerte (de 9 a 1-3).
• sources: cst-only como default global: más rápido pero pierde foro y espacolegislacao para queries funcionales.

Estrategia lang (benchmark 2026-05-15):

Señales para detectar query ES: keywords como configuración, cuentas, página, solución, o presencia de ñ / acentos en posición de keyword español. Ver docs/benchmark-results/summary-multilang.md.

What's done vs what's planned

✅ Implemented (v0.5.0)

• All 5 phases of the original plan + v0.5 roadmap: CST aggregator, all adapters, fan-out engine, fetch router, YouTube transcripts, Brave Search, MCP server, 5 MCP tools, audit log, protheus_query, schema SSOT, --schema flag, po_ui multi-word scoring.
• 22+ Go packages, ~160 tests, both binaries build clean.
• MCP protocol 2025-06-18, tool annotations, output schemas.
• Bug A + B fixed (see docs/known-issues/, EVALUATION.md).

📋 Planned but NOT implemented

• Zendesk authenticated mode (community posts + private articles). Blueprint in docs/zendesk-auth-playwright-plan.md. pkg/zendesk.Client.WithCookie() is ready; what's missing is pkg/zendesk/playwright.go and internal/cli/auth_zendesk.go. Estimated 1 week. Trigger: 3+ users reporting they need community posts.

🟡 Code paths that exist but were never E2E-smoked

• mapj fetch youtube:<id>: works but the dev machine didn't have yt-dlp installed during initial build. Tests use a fake runner. To smoke-test: winget install yt-dlp, then mapj fetch youtube:<real-id>.
• mapj search --include-web: works but no real Brave API key was available. Tests use httptest mocks. To smoke-test: set BRAVE_API_KEY and run with --include-web.

Building & testing

# Build both binaries
go build -o mapj.exe ./cmd/mapj
go build -o mapj-mcp.exe ./cmd/mapj-mcp

# Run the full suite
go test ./...
go vet ./...

# Coverage for the new packages
go test -cover ./pkg/cst/... ./pkg/search/... ./pkg/fetch/... ./mcp/...

On Windows / PowerShell: avoid 2>&1 with Go subcommands — PowerShell 5.1 wraps stderr in ErrorRecord and reports false failure when the binary actually succeeded. Use Bash (Git Bash) for piping into the MCP server.

Memory & external context

Persistent memory for this project lives at: %LOCALAPPDATA%\anthropic\Claude\projects\D--TOTVS-development-area-agentes-tools-totvs-IA\memory\

Relevant notes for future agents:

• mapj_phases_complete.md — high-level status of the project.
• cst_busca_endpoint.md — auth-free aggregated search endpoint details.
• zendesk_api_facts.md — Zendesk endpoints, what auth they need.
• search_apis_landscape.md — why Brave and not Google CSE.
• known_bugs.md — bugs in the OTHER two Python tools (Kimi, Agentes_totvs) that we replaced.

Anti-patterns — please don't

• ❌ Don't add fmt.Println(...) to library code; use zap.L().Debug/Info/....
• ❌ Don't create new top-level commands — extend search or fetch if it fits semantically. The CLI surface should stay compact.
• ❌ Don't reintroduce the --debug/--dump-debug flags (removed in v0.2.0). Use --log-level debug for diagnostics.
• ❌ Don't add a CSV or "human pretty" output format. Removed in v0.2.0 on purpose — there are only 4 formats supported.
• ❌ Don't introduce JSON-RPC SDKs. The mcp/ package is intentionally hand-rolled to keep the dependency surface minimal.
• ❌ Don't try to bypass captchaEnabled+WAF+2FA programmatically. The only sane path for Zendesk auth is docs/zendesk-auth-playwright-plan.md.

Quick checks before commit

go fmt ./...
go vet ./...
go test ./...
go build -o mapj.exe ./cmd/mapj
go build -o mapj-mcp.exe ./cmd/mapj-mcp

If any of those fail, do not commit.

Useful entry points by task

Good luck.