bqnlsp

Imported from Grshor/bqnlsp (AGENTS.md). Install upstream with npx skills add Grshor/bqnlsp. Copyright stays with the author.

Repository Guidelines

Language Server Protocol server for BQN. Rust workspace; upstream home is https://sr.ht/~detegr/bqnlsp/. Licensed GPL-3.0 (LICENSE) — the BQN/ submodule is separately ISC; do not treat the MIT header in editors/vscode/src/extension.ts as this repo's license.

Architecture & Data Flow

The server does not implement BQN. It embeds BQN's own self-hosted compiler (BQN/src/c.bqn, BQN/src/glyphs.bqn) via include_str! and drives it through the cbqn crate.

did_open / did_change (text, FULL sync)
  └─ bqn::compile(code)            lsp/src/bqn.rs   → BQNResult (bytecode: Vec<u64>, locs, blocks, bodies, constants)
       ├─ infer::check(&compiled)  lsp/src/infer.rs → abstract interpretation → JSON report
       └─ diagnostics::get_diagnostics(&BQNResult)  lsp/src/diagnostics.rs
  └─ DashMap.insert(uri, DocumentData) then client.publish_diagnostics

Per-request features read the cached CompilerResult and decode bytecode lazily: highlight.rs (highlight / rename / references / definition / symbols), infer::hover_at, help_for_symbol.

Layering — respect it when editing:

Key invariants:

• All server state is one DashMap<Url, DocumentData> (lsp/src/main.rs:28), where DocumentData = (Vec<String>, BQNResult, serde_json::Value) = (source lines, compile result, inference report). No Mutex/RwLock, no tokio::spawn, no debouncing, no version checking — every did_change recompiles and re-infers from scratch on the calling tokio worker.
• #[cfg(feature = ...)] does not exist in lsp/src. native/wasi only select the cbqn backend; platform conditionals live solely in lsp/build.rs (rpath).
• Cross-module results travel as opaque serde_json::Value (the inference report) so hover and diagnostics never re-run analysis. infer::check, infer::report_diagnostics, infer::hover_at and main.rs:format_infer_hover all speak that contract.
• Lenient-by-design fallback. Unknown opcode, primitive, or glyph yields Abs::Unknown / None rather than a diagnostic (contracts.rs, infer.rs). Never produce a false positive; preserve this when adding coverage.

Key Directories

Development Commands

Prerequisites (all mandatory for a working build):

git submodule update --init --recursive   # else include_str! of BQN/src/{glyphs,c}.bqn fails
# CBQN must be built separately: `make shared-o3` in a CBQN checkout (native feature links libcbqn)
# first build also needs network: cbqn / tower-lsp are not vendored

Build and run:

./build.bqn /path/to/CBQN                 # canonical: cargo build --release --bin genhelp
                                          #         → cargo run --bin genhelp ./BQN ./lsp/src/help
                                          #         → cargo build --release --bin bqnlsp
RUSTFLAGS="-L /path/to/CBQN" LD_LIBRARY_PATH="/path/to/CBQN" cargo build --release --bin bqnlsp   # manual
cargo run --release --bin genhelp ./BQN ./lsp/src/help   # regenerate help (required, see below)
nix --extra-experimental-features 'nix-command flakes' build 'sourcehut:~detegr/bqnlsp'   # → result/bin/bqnlsp

Agent-facing non-LSP entry points (lsp/src/main.rs):

bqnlsp                      # speaks LSP on stdin/stdout (what editors run)
bqnlsp check FILE...        # text summary; exit 1 when a file has problems, 2 on bad use
bqnlsp check --json FILE    # the machine report, same exit statuses
bqnlsp symbols FILE         # the document outline, nested
bqnlsp probe FILE           # every bytecode body/instruction with source substrings
bqnlsp --help

--check and --probe are the older spellings and still work unchanged — including --check's always-zero exit and JSON output, which is the published machine interface and what the corpus asserts on.

Verified working recipe on this machine (deps are not vendored; a usable CBQN shared object already exists in a sibling checkout):

cargo fetch                                                   # once, needs network
RUSTFLAGS="-L $HOME/Projects/finance" cargo build --bin bqnlsp
LD_LIBRARY_PATH=$HOME/Projects/finance target/debug/bqnlsp --check file.bqn

Cold build ≈11 s — the native feature does not pull in wasmer/cranelift. --check exits 0 even when compilation or inference fails; assert on the JSON, not the exit status (exit 2 means only bad arguments or an unreadable file).

--check is documented in-code as "the agent-facing entry point" and emits {compile_ok, diagnostics[], constraints[], signatures[], trace[]} (or {compile_ok:false, error, range, ...} on a compile error). Prefer it over the LSP protocol for verifying interpreter or bytecode work.

No formatter, linter, or toolchain config is committed (no rustfmt.toml, clippy.toml, .editorconfig, rust-toolchain.toml) — defaults apply; stable toolchain.

Code Conventions & Common Patterns

Error handling — three deliberate regimes:

• Result<_, String> where the String is a user-visible message (contracts.rs, infer.rs); Result for FFI-facing code (bqn::compile_impl -> Result<BQNResult, cbqn::Error>).
• Option = "not applicable" for every request implementation (*_impl in main.rs, highlight::get_variables).
• unwrap() / expect() / panic! only where an invariant is broken and crashing is the intended response: bytecode decoding (bqn.rs ~20 sites, panic!("bad block case"), panic!("bad error rank")), bodies.find(..).unwrap() in infer.rs. Never use them for user-input validation.
• No logging framework — eprintln! only. cbqn::Error is surfaced once as BQNResult::InternalError; user syntax errors arrive as data through BQN's ⎊ guard, becoming BQNResult::Error with source spans.

Do these things:

• Cache process-global pure computations in OnceLock (COMPILER in bqn.rs, OPCODE_ARGN in bytecode.rs, GLYPH in infer.rs, PositionConverter::line_info).
• Hold the DashMap Ref guard from self.documents.get(&uri) for the whole request computation; there is no lock ordering to worry about.
• Add a primitive by extending the match arms in contracts.rs (prim_monadic / prim_dyadic, plus derive_mod1 / derive_mod2 for modifiers) and, if it changes shapes, the helpers in shape.rs. Unsupported glyphs must fall through to Abs::Unknown.
• Add hover help by adding a match arm in lsp/src/help/mod.rs plus a matching generated .md.
• Do all identifier work through highlight::get_variables + utils::{bqn_name_info, convert_alike} for rename/highlight — never string-match BQN names by hand.
• Add bytecode opcodes in bytecode.rs alongside the Opcode constants and opcode_argn's arity table; probe real output with bqnlsp --probe.
• Treat codepoint indices as the internal currency (CompilerResult.locs are codepoint ranges) and convert at the LSP boundary with utils::PositionConverter (UTF-16 in/out). Non-BMP BQN glyphs (𝕩, 𝕨, 𝔽, …) are single chars with len_utf16() == 2 and need no special-casing.

Gotchas:

• lsp/src/help/*.md are gitignored and untracked, yet include_str! in lsp/src/help/mod.rs requires all ~96 to exist. A fresh clone cannot cargo build until genhelp has run (or stubs are written).
• fs_extra is a declared but unused build-dependency of bqnlsp; lsp/build.rs is 12 lines and only emits rpath args + BQN_PATH.
• BQNLSP_BQN_PATH (default ../../BQN/) feeds cargo:rustc-env=BQN_PATH, consumed by include_str! in bqn.rs.
• genhelp depends on cbqn without default-features = false, so it always needs native libcbqn even in a wasi-only workflow.

Important Files

Runtime/Tooling Preferences

• Cargo workspace, resolver = "2", members ["lsp", "genhelp"]. Both crates: edition = "2021", rust-version = "1.77". No rust-toolchain.toml; Nix pins rust-bin.stable.latest.default.
• Feature matrix: default = ["native"] → cbqn/native-backend links libcbqn.{so,dylib,dll} (needs a C toolchain + CBQN). wasi → cbqn/wasi-backend runs the committed lsp/BQN.wasm (1.2 MB) through wasmer; no C toolchain. The wasi path needs BQN_WASM set at compile time (the cbqn crate include_bytes!s it) — .cargo/config.toml currently points it at a stale Windows path.
• Runtime linking: lsp/build.rs injects $ORIGIN (Linux) / @loader_path (macOS) so libcbqn is found next to the binary; otherwise LD_LIBRARY_PATH / DYLD_LIBRARY_PATH. Ship libcbqn alongside the installed binary.
• Nix is the preferred/reproducible path (pkgs.cbqn, BQN fetched at a pinned rev, genhelp run in lsp's preBuild). build.bqn is the non-Nix one-shot.
• editors/vscode is a TypeScript/npm project (tsc -b, eslint); it is a dev-only template — no publisher, no packaging.

Testing & QA

There are zero Rust tests and no CI. No #[test] / #[cfg(test)] / #[tokio::test] anywhere in lsp/ or genhelp/; no tests/, examples/, benches/; no [dev-dependencies] in any manifest; no .github/ or .builds/ at the repo root, and no .yml/.yaml file anywhere in the checkout (root or submodule). cargo test compiles 0 tests. The flake sets cargoTestOptions but nothing runs.

Practical verification, in order of preference:

0. cargo test -p bqnlsp — two integration tests: corpus.rs (golden --check reports, 35 cases) and cli.rs (commands, output and exit statuses). Both spawn the real binary and skip loudly without CBQN.
1. bqnlsp --check <file.bqn> or bqnlsp check <file.bqn> — the sanctioned surface: assert on the JSON report (compile_ok, diagnostics, constraints, signatures, trace). Use throwaway .bqn inputs.
2. bqnlsp --probe <file.bqn> — dump bytecode bodies/instructions when changing bytecode.rs or infer.rs.
3. Manual editor session via editors/neovim/nvim-lspconfig/bqnlsp.lua or the VS Code extension — the only end-to-end path; there is no mock transport or LSP harness, because Backend::new requires a real tower_lsp::Client and changed_document publishes diagnostics.

Notes for anyone adding tests: a test binary still links CBQN natively, so bare cargo test fails without RUSTFLAGS=-L <cbqn>/libcbqn present — which is why the --check CLI is the pragmatic harness. Pure, client-free modules (bytecode.rs, utils.rs, shape.rs, contracts.rs, infer.rs) are the natural first targets.

The BQN-language suite in the submodule is not this project's coverage — it tests the BQN language. Run it only when validating CBQN/submodule behavior:

cd BQN && bqn test/this.bqn            # all cases/ files
bqn test/this.bqn simple syntax        # selected case files
bqn test/this.bqn -noerr               # skip expected-error cases

Case format (BQN/test/cases/*.bqn): expected % expression; ! % expression = must fail; a bare line with no % must evaluate to 1; # starts a comment. BQN/test/unit.bqn reruns the same cases against repo components (-nocomp, -rt, -ref).

Graft Context Graph

graft/ holds a local, regenerable context graph: one markdown card per source file (every symbol with its file:line span) plus graft/.graph/wiring.json, the tree-sitter call graph. Current graph: 15 files, 260 nodes, 269 edges, indexed in [lua, nix, rust, typescript]. It is gitignored (/graft/ in .gitignore) and .ignore re-admits it to ripgrep; each teammate builds their own.

graft build          # rebuild (~0.25s here); --deep adds the LLM concept/summary pass
graft check          # exit 1 if the graph drifted from the code
graft map            # token-budgeted orientation: directory clusters, hubs, hotspots
graft skeleton lsp/src/infer.rs     # every signature in a file, no bodies
graft callers get_variables         # who calls it (--direction out for callees, --depth N transitively)
graft grep "<regex>"                # exhaustive, grouped by enclosing symbol
graft ask "<task>" --source         # ranked nodes with code inlined
graft blast                         # blast radius of the working-tree diff

Use it for the where and how it is wired — cheapest for call-graph questions, file API surfaces, and blast radius, with no language server required. Do not use it for the why: symbols and edges carry no intent, so this file (and README.md, BQN/*.md) stays authoritative for design and policy.

Limits here:

• BQN/ is not indexed (gitignored, and BQN is not a supported language) — ask graft nothing about the submodule; read it directly.
• Edges are tree-sitter approximations, not compiler-grade. For rename/definition work use OMP lsp; for rewrites use ast_grep / ast_edit; for ranges use read path:LINE-LINE.
• Ranked output is top-N, never exhaustive — for "find every …" use graft grep.
• The deep tier is unbuilt; graft check reports every node as pending-meaning. That is expected, not drift.
• Queries re-sync automatically, so graft build is only needed after a change large enough to matter.

graft init has been run here, repo-local only: it wrote opencode.json (mcp.graft) and appended a fenced <!-- graft:start --> … <!-- graft:end --> block at the end of this file. No machine-wide config was touched (no ~/.claude/*, ~/.codex/*, .mcp.json). Running it without --no-global would additionally write ~/.claude/settings.json hooks, ~/.claude.json, ~/.codex/config.toml, and ~/.codex/hooks.json — machine-wide, affecting every repo.

Graft — repo context graph

This repo is indexed in graft/: small linked markdown nodes that explain each system and carry exact file:line spans, kept in sync with the code through git.

For ANY task here — understanding how something works, finding where code lives, or scoping a change — get context from the graph before grepping or opening source files. Re-ask freely (it's cheap) and reuse literal identifiers you already have (symbol, error string, file name) as the query. New to this repo? Run graft map first — a token-budgeted orientation (dir clusters, hubs, hotspots), no LLM, no key.

• Run graft ask "<your question>" --source → ranked nodes with the relevant code spans inlined (each hit's ≤8-line crux by default; --full for whole definitions when the crux isn't enough). Match the tool to the task shape: for understanding or editing, the top node IS the answer — cite its covers: file:line spans and edit straight from --source. For exhaustive tasks ("every occurrence / every caller of this pattern"), ranked results are top-N, not complete — run graft grep "<literal>" instead (exhaustive over indexed files, grouped by enclosing symbol), falling back to raw grep -rn only for unindexed files.
• graft skeleton <file> → every definition's signature + span, ~10× cheaper than reading the file; use it to skim an API surface.
• graft callers <symbol> gives precomputed, exact edges — who calls this. Add --direction out for what it calls, or --depth N to walk transitively for the full blast radius. For structural questions, skip ranking and use this directly.
• Or browse: graft/INDEX.md lists every node; follow the links.
• Monorepos and folders of multiple repos rank fairly across sub-projects — hits carry [scope/] labels naming which one they're from. Narrow with graft ask "<task>" --in <scope>/ once you know where you're working.

If a returned span is truncated ("+N more lines"), open the file at that exact range before finalizing. Only open source files when a node genuinely lacks a needed detail, and then at the exact file:line the node points to — never re-read whole files.

After big code changes, refresh the graph with graft build (deterministic, no API key, $0).