Imported from chuchitrii/recutils-js (AGENTS.md). Install upstream with npx skills add chuchitrii/recutils-js. Copyright stays with the author.

AGENTS.md

Project Overview

This is a JavaScript/WebAssembly wrapper for the GNU recutils library. The project brings recutils functionality to JavaScript environments (Node.js, browsers, Obsidian plugins) by compiling the original C library to WebAssembly.

Approach

We don't reimplement recutils - we reuse the original.

Primary approach: Compile original GNU recutils C code to WebAssembly
Minimal custom C code: Only write new C when original code cannot compile to WASM
Maximum compatibility: Leverage decades of GNU recutils development and testing
Faithful behavior: Maintain identical functionality, argument parsing, and output formatting
Strategic patching: Apply minimal patches only where WASM environment requires it

Current Status

✅ COMPLETED FEATURES:

7 GNU recutils utilities (recsel, recins, recdel, recset, recfmt, recinf, recfix)
Full CLI compatibility with all GNU recutils options
JavaScript API with multi-value field support
WASM-based parse() method with native GNU recutils parsing
Official test suite compatibility (336 of 342 tests passing)

Project Structure

Directory Layout

recutils-js/
├── dist/                   # Build output for npm package
│   ├── modules/           # Compiled WASM modules
│   │   ├── recsel.js     # WASM loader for recsel
│   │   ├── recsel.wasm   # Compiled recsel WebAssembly
│   │   └── ...           # Other utility modules
│   ├── recsel/           # recsel distribution files
│   │   ├── index.js      # ESM module entry
│   │   ├── index.cjs     # CommonJS entry
│   │   ├── cli.js        # CLI executable
│   │   └── wasm.js       # WASM wrapper
│   └── ...               # Other utilities follow same pattern
│
├── src/                    # Source code
│   ├── shared/           # Shared utilities
│   │   ├── recutils_core.c  # Core C functions
│   │   └── cli-utils.js     # CLI helper functions
│   ├── modules/          # Built WASM modules (gitignored)
│   ├── recsel/           # recsel implementation
│   │   ├── recutils_recsel.c  # C implementation
│   │   ├── index.js          # Three-tier API
│   │   ├── cli.js            # CLI entry point
│   │   └── wasm.js           # WASM wrapper
│   └── ...               # Other utilities follow same pattern
│
├── gnu-recutils/          # Git submodule - original GNU recutils
│   ├── src/              # GNU recutils C source
│   ├── torture/          # Official test suite
│   └── doc/              # Documentation
│
├── scripts/              # Build and test automation
│   ├── build-recutils.sh    # Build GNU recutils with Emscripten
│   ├── build.sh             # Build WASM modules
│   ├── build-dist.sh        # Prepare npm distribution
│   ├── test.sh              # Run official test suite
│   └── patches/             # Patches for WASM compatibility
│
├── tests/                # JavaScript test files
│   ├── recsel.test.js   # Unit tests for recsel
│   ├── integration.test.js  # Integration tests
│   ├── multiline.test.js    # Multiline field tests
│   └── ...
│
├── examples/             # Example applications
│   ├── exercise-notes/  # React app with recutils
│   └── cli-interactive/ # Interactive CLI interface
│
├── package.json         # NPM package configuration
├── README.md           # Main documentation
├── AGENTS.md           # This file - AI guidance
├── DEVELOPMENT.md      # Build instructions
├── UNIMPLEMENTED-FEATURES.md  # Known limitations
└── ROADMAP.md          # Future development plans

File Naming Conventions

C Files: recutils_<utility>.c - WASM implementation
JS Modules: index.js - Main API entry point
CLI Scripts: cli.js - Command-line interface
WASM Wrappers: wasm.js - Low-level WASM interface
Tests: <utility>.test.js - Jest test files
Scripts: <action>-<target>.sh - Shell scripts

Architecture

Architecture Layers

The project uses a modular architecture with separate folders for each utility. Each utility provides 3 consumption methods:

WASM Core Layer (src/shared/recutils_core.c + src/*/recutils_*.c)
- Shared C core functions and utility-specific implementations
- Direct wrappers around GNU recutils library functions
- Functions: recutils_select_argv(), recutils_parse_json(), recutils_insert(), etc.
- Memory management and error handling for WASM environment
WASM Wrapper Layer (src/*/wasm.js)
- Thin JavaScript wrapper around each utility's WASM functions
- Handles WASM memory allocation and argument marshaling
- Direct interface to compiled utility modules
Three-Tier API Layer (src/*/index.js)
- JS Options API: recsel(data, { expression: 'Age > "25"' })
- String Args API: recselString(data, '-e "Age > 25" --count')
- WASM Instance API: getWASMInstance() for direct access
CLI Layer (src/*/cli.js)
- Command-line scripts: recsel-js, recins-js
- 100% compatible with original GNU recutils CLI
- Uses modular WASM builds for correct functionality

API Design

High-Level Functional API (recommended for most users)

const { recsel, recins, parse } = require('recutils');
const filtered = await recsel(data, { expression: 'Age > "25"' });
const updated = await recins(data, { fields: { Name: 'John' } });
const records = await parse(filtered);

String Arguments API (CLI-like but in JavaScript)

const { recselString, recinsString } = require('recutils');
const result = await recselString(data, '-e "Age > 25" --count');
const updated = await recinsString(data, '-f Name -v John');

Direct Utility Import (modular access)

const { recsel } = require('recutils/recsel');
const { recins } = require('recutils/recins');
const result = await recsel(data, { expression: 'Age > "25"' });

Command-Line Interface (full GNU recutils compatibility)

recsel-js -e 'Age > "25"' -p Name,Age data.rec
recins-js -f Name -v John -f Age -v 30 data.rec

Adding New Utilities

All GNU recutils utilities are already implemented. However, if a new utility were added to GNU recutils in the future, follow this systematic approach:

Step 1: Create Utility Folder

mkdir src/newutil

Step 2: Add C Implementation (`src/newutil/recutils_newutil.c`)

#include "../shared/recutils_core.h"

EMSCRIPTEN_KEEPALIVE
char* recutils_newutil(const char* input_data, char** argv, int argc) {
    // Use shared argument parsing logic
    parse_argv_with_original_logic(argc, argv);
    
    // Implement utility-specific logic
    // Follow the same pattern as other utilities
    
    return output;
}

Step 3: Add WASM Wrapper (`src/newutil/wasm.js`)

const NewutilModule = require('../../dist/modules/newutil.js');

class NewutilWASM {
    // Follow the same pattern as other WASM wrappers
    // Implement callWithArgv method
}

Step 4: Add Three-Tier API (`src/newutil/index.js`)

/**
 * Three-tier API for newutil
 */
async function newutil(inputData, options = {}) {
    // JS options object API
}

async function newutilString(inputData, argString = '') {
    // String arguments API
}

module.exports = { newutil, newutilString, getWASMInstance };

Step 5: Add CLI Script (`src/newutil/cli.js`)

#!/usr/bin/env node
// Follow the same pattern as other CLI scripts
// Handle all command-line arguments and options

Step 6: Update Package Configuration (`package.json`)

{
  "bin": {
    "recsel-js": "src/recsel/cli.js",
    "recins-js": "src/recins/cli.js", 
    "newutil-js": "src/newutil/cli.js"
  },
  "exports": {
    "./newutil": "./src/newutil/index.js"
  }
}

Step 7: Add Tests (`scripts/test.sh`)

# Copy test file
cp -r "$OLDPWD/gnu-recutils/torture/utils/newutil.sh" .

# Create wrapper script  
cat > newutil << 'EOF'
#!/bin/bash
exec node src/newutil/cli.js "$@"
EOF
chmod +x newutil

# Update test patterns to include newutil-*
grep -E "  (recsel|recins|newutil)-.*(ok|fail|error)" test_output.log

Development Conventions

Code Style

Language: JavaScript with JSDoc annotations (no TypeScript)
Comments: Avoid obvious comments - code should be self-documenting
- ❌ Bad: // Increment counter before counter++
- ❌ Bad: // Check if user exists before if (user)
- ✅ Good: Comments explaining WHY, not WHAT
- ✅ Good: Complex algorithm explanations
- ✅ Good: Workarounds with context
Testing: Test against original GNU recutils test suite for compatibility
Automation: Always update scripts instead of manual fixes for reproducibility
Build: Use npm scripts - all build logic is in scripts/ directory
Git Strategy: Use git submodules for original recutils repository
IMPORTANT: Always use npm run build, npm run test, etc. Never run scripts directly with ./scripts/

Testing Against Original GNU Recutils

When implementing or debugging functionality, always compare against the original GNU recutils:

# Install original GNU recutils for comparison testing
brew install recutils  # macOS
apt-get install recutils  # Ubuntu/Debian

# Compare behavior directly
echo "Name: Alice\nAge: 25" > test.rec

# Test original
recsel -e 'Age > "20"' test.rec
recins -e 'Name = "Alice"' -f Status -v Updated test.rec

# Test our implementation  
recsel-js -e 'Age > "20"' test.rec
recins-js -e 'Name = "Alice"' -f Status -v Updated test.rec

# Output should be identical

This approach helps ensure:

Exact compatibility with GNU recutils behavior
Correct argument parsing and option handling
Proper output formatting and record structure
Error handling that matches original utilities

Build System

See DEVELOPMENT.md for detailed build instructions, troubleshooting, and system requirements.

Key Commands for AI Development Work

npm run build - Compile WASM modules after code changes (2-3 min)
npm run test:official - Run against GNU recutils test suite (compatibility verification)
npm run build-recutils - Full rebuild of GNU recutils libraries (20-25 min, rarely needed)

Build System Philosophy

Automated patching: All compatibility fixes are automated in scripts
Reproducible builds: Update scripts, never do manual fixes
Incremental builds: Most changes only require npm run build
GNU compatibility: Maintain exact compatibility with original recutils behavior

Testing Strategy

Two-Tier Testing Approach

Tier 1: CLI Compatibility Testing

Method: Test our CLI utilities against the official GNU recutils test suite
Command: npm run test:official
Purpose: Ensure 100% behavioral compatibility with original GNU recutils
Coverage: 336/342 testable tests passing (98.2% compatibility, 6 expected failures)
Scope: All CLI options, argument parsing, output formatting, edge cases

Tier 2: JavaScript API Testing

Method: Test JavaScript API functions against our own CLI implementations
Command: npm run test:js
Purpose: Verify that JS APIs produce identical results to CLI utilities
Coverage: Comprehensive unit tests for all utilities and edge cases
Scope: Options parsing, multi-value fields, error handling, multiline support

Testing Philosophy

CLI as Source of Truth: Our CLI utilities must match GNU recutils exactly
JS API Consistency: JavaScript functions must match our CLI behavior
Transitive Compatibility: If CLI matches GNU and JS matches CLI, then JS matches GNU
Comprehensive Coverage: Test both happy paths and error conditions
Cross-Implementation Verification: Same inputs produce identical outputs across all layers

Current Status

GNU Compatibility: 98.2% (336/342 tests passing, 6 expected failures)
JavaScript Tests: All major functionality covered including multiline fields, expressions, aggregation
Expected Failures: 4 encryption tests, 1 external descriptor test, 1 template formatting test (see UNIMPLEMENTED-FEATURES.md)

Important Notes

DO (AI Development Guidelines):

✅ Test against official GNU recutils: Always compare behavior with original GNU recutils
✅ Update scripts for reproducibility: Automate fixes instead of manual changes
✅ Use WASM core for logic: Put complex processing in C, keep JavaScript thin
✅ Follow modular architecture: Each utility has separate C/JS/CLI layers
✅ Maintain GNU compatibility: API should match original recutils behavior exactly
✅ Use build commands properly: npm run build for most changes, avoid long rebuilds

DON'T (AI Development Anti-patterns):

❌ Manual fixes: Always update automation scripts instead
❌ Break GNU compatibility: Users expect identical behavior to original recutils
❌ Complex JavaScript logic: Move heavy processing to WASM core
❌ New documentation files: Consolidate into existing docs (README, DEVELOPMENT, AGENTS, UNIMPLEMENTED)
❌ Timeout on build-recutils.sh: This script can take 20-30 minutes legitimately

Current Priorities

Maintenance: Keep build system working and all tests passing
Documentation: Maintain comprehensive guides and examples
Performance: Continue leveraging WASM for optimal performance
Ecosystem: Support for various JavaScript environments (Node.js, browsers, Obsidian)
Stability: Focus on bug fixes and edge case handling over new features

Documentation Structure

README.md: Project overview, installation, basic usage examples
AGENTS.md (this file): AI assistant guidance, architecture, development patterns
DEVELOPMENT.md: Build system, prerequisites, troubleshooting, contributing
UNIMPLEMENTED-FEATURES.md: Known limitations and missing features
ROADMAP.md: Future development plans and priorities

Useful References

GNU recutils manual - Original recutils documentation
Official test suite - Comprehensive compatibility tests
Emscripten docs - WebAssembly compilation reference

recutils-js

About

AGENTS.md

Project Overview

Approach

Current Status

Project Structure

Directory Layout

File Naming Conventions

Architecture

Architecture Layers

API Design

Adding New Utilities

Step 1: Create Utility Folder

Step 2: Add C Implementation (`src/newutil/recutils_newutil.c`)

Step 3: Add WASM Wrapper (`src/newutil/wasm.js`)

Step 4: Add Three-Tier API (`src/newutil/index.js`)

Step 5: Add CLI Script (`src/newutil/cli.js`)

Step 6: Update Package Configuration (`package.json`)

Step 7: Add Tests (`scripts/test.sh`)

Development Conventions

Code Style

Testing Against Original GNU Recutils

Build System

Key Commands for AI Development Work

Build System Philosophy

Testing Strategy

Two-Tier Testing Approach

Testing Philosophy

Current Status

Important Notes

DO (AI Development Guidelines):

DON'T (AI Development Anti-patterns):

Current Priorities

Documentation Structure

Useful References

Use it

Manifest

Reviews

About

AGENTS.md

Project Overview

Approach

Current Status

Project Structure

Directory Layout

File Naming Conventions

Architecture

Architecture Layers

API Design

Adding New Utilities

Step 1: Create Utility Folder

Step 2: Add C Implementation (src/newutil/recutils_newutil.c)

Step 3: Add WASM Wrapper (src/newutil/wasm.js)

Step 4: Add Three-Tier API (src/newutil/index.js)

Step 5: Add CLI Script (src/newutil/cli.js)

Step 6: Update Package Configuration (package.json)

Step 7: Add Tests (scripts/test.sh)

Development Conventions

Code Style

Testing Against Original GNU Recutils

Build System

Key Commands for AI Development Work

Build System Philosophy

Testing Strategy

Two-Tier Testing Approach

Testing Philosophy

Current Status

Important Notes

DO (AI Development Guidelines):

DON'T (AI Development Anti-patterns):

Current Priorities

Documentation Structure

Useful References

Use it

Manifest

Reviews

Step 2: Add C Implementation (`src/newutil/recutils_newutil.c`)

Step 3: Add WASM Wrapper (`src/newutil/wasm.js`)

Step 4: Add Three-Tier API (`src/newutil/index.js`)

Step 5: Add CLI Script (`src/newutil/cli.js`)

Step 6: Update Package Configuration (`package.json`)

Step 7: Add Tests (`scripts/test.sh`)