Imported from TekneGram/dialogitech (
AGENTS.md). Install upstream withnpx skills add TekneGram/dialogitech. Copyright stays with the author.
AGENTS.md
Purpose
This repository processes academic papers into structured, classified chunk data and stores those chunks in LanceDB for later research retrieval.
The pipeline is:
- start from a PDF or an existing Marker JSON artifact
- extract paper metadata from Marker JSON
- convert Marker JSON into filtered Markdown
- remove the
Referencessection and everything after it - chunk the remaining paper body by heading
- classify chunks as
abstract,introduction,method,results, ordiscussion - use Gemma 4 fallback for unresolved chunks
- generate local embeddings with Ollama
- insert chunks plus metadata into LanceDB
The key architectural rule is that incomplete classification is not acceptable. Unresolved chunks must not be inserted into the database.
Key Directories
chunker/: active project code for metadata extraction, filtering, chunking, and classificationdbinsert/: LanceDB schema, embedding, ingestion, indexing, and full-pipeline orchestrationdbquery/: query rewriting, retrieval, ranking, and summarization over LanceDBdbxquery/: grounded follow-up querying over prior synthesized summariesresquery/: iterative research sessions over LanceDB using prior claims as turn contextmarker/: local Marker checkout plus conversion outputspdfs/: source PDFs
Core Modules
chunker/metadata_extractor.py
Extracts:
- title
- authors
- journal metadata
- references
from Marker JSON.
chunker/boilerplate_filter.py
Converts Marker JSON into filtered Markdown.
It should:
- remove front matter noise
- remove figures, tables, page furniture, and boilerplate
- remove the
Referencessection - drop everything after
References, including appendices, prompt templates, and supplementary content - preserve the paper body heading structure
This references cutoff is important. Appendix material must not reach chunking or classification.
chunker/markdown_section_chunker.py
Splits filtered Markdown into heading-based chunks using:
MarkdownHeadingSectionChunkHeadingSplitMarkdownSectionChunker
Chunks are word-count based with overlap and sentence-aware boundaries. The overlap logic should not start a new chunk in the middle of a sentence.
chunker/chunk_models.py
Pipeline-facing classification layer. It should contain only:
ChunkClassificationClassifiedSectionChunkClassifiedHeadingSplitDeterministicSectionClassifierChunkClassificationEnricherclassify_filtered_markdown(...)
Responsibilities:
- deterministic heading-based rules
- orchestration between deterministic classification and the LLM classifier
- passing the previous resolved label into the LLM edge-case path when needed
Do not move prompt logic, markdown anchoring, JSON parsing, or MLX runtime control back into this file.
chunker/llm_section_classifier.py
LLM-specific classification layer. It should contain:
ChunkClassificationLLMChunkLocationChunkContextLLMDecisionarticle_quintile(...)
Responsibilities:
- prompt construction
- quintiles
- chunk anchoring inside filtered Markdown
- context retrieval
- response parsing and recovery from malformed JSON-like model output
- external MLX subprocess integration
- chunk-level logging for requests and responses
The current controller behavior is:
- classify the chunk directly
- if needed, allow one context round
- if the model requests context again, send one final constrained prompt using the previous resolved chunk label and require a concrete label
If Gemma still does not provide a concrete label but a previous resolved chunk label exists, the classifier should log the failure and deterministically inherit the previous resolved label. If no previous resolved label exists, the run should still fail rather than silently continue.
chunker/mlx_llm_runner.py
Thin external runner for Gemma 4 when using the separate Unsloth MLX environment.
Reads JSON from stdin and writes raw model output to stdout.
chunker/run_section_classification.py
CLI runner for classifying a filtered Markdown file from the terminal.
dbinsert/full_pipeline_service.py
Main end-to-end pipeline orchestration from PDF to LanceDB.
Current behavior:
- derives
paper_idfrom the PDF filename - reuses Marker JSON unless rerun is forced
- can rerun filtered markdown and classification independently
- writes per-paper Marker and classification logs
- requires all chunks to be resolved before DB insertion
dbinsert/run_full_pipeline.py
CLI entry point for the full PDF-to-LanceDB pipeline.
dbinsert/embedding_service.py
Embedding backends.
The practical default is local Ollama embeddings with:
- model:
qwen3-embedding:0.6b
dbinsert/index_manager.py
Creates LanceDB vector, full-text, and scalar indexes.
Index creation must be idempotent. Repeated ingests should not fail because an index already exists.
dbquery/
Database-query code should stay modular and split by responsibility.
Preferred shape:
- one main class per file
- one concern per class
- orchestration in a pipeline/coordinator class
- prompt construction and response parsing in the LLM-facing class that owns that prompt
- database retrieval in a retrieval-specific class
- ranking/fusion in a ranking-specific class
- output rendering in a writer-specific class
If logic starts getting long:
- move shared helpers into small modules under
dbquery/utils/ - keep helpers narrow and reusable
- avoid growing one file into a mixed retrieval-plus-prompt-plus-output module
For dbquery, prefer small files and explicit dependency injection over large utility-heavy base classes.
dbxquery/
Follow-up query code should stay modular and split by responsibility.
Preferred shape:
- one main class per file
- one concern per class
- orchestration in a follow-up pipeline/coordinator class
- evidence planning in an LLM-facing planner class
- plan validation in a validator class
- evidence retrieval in a retrieval adapter class
- grounded answer writing in an LLM-facing writer class
- shared parsing or prompt helpers in
dbxquery/utils/
Current scope:
- single-turn only
- synchronous only
- follow-up input is
prior synthesized summary + user question - Gemma emits a strict JSON evidence plan
- the plan may call only one tool:
retrieve_evidence - the final answer must be written from retrieved evidence only
Allowed follow-up filters are:
paper_id_inauthors_anyyear_minyear_maxclassification_label_insection_title_contains
dbxquery should prefer the ## Synthesized Summary section from a prior dbquery output artifact when present, rather than reusing the entire prior query markdown file as prompt context.
resquery/
Iterative research-session code should stay modular and split by responsibility.
Preferred shape:
- one main class per file
- one concern per class
- orchestration in a session pipeline/coordinator class
- query-context construction in a dedicated builder class
- claim extraction in a dedicated LLM-facing extractor class
- follow-up suggestion generation in a dedicated LLM-facing suggester class
- session persistence in a state store class
- output rendering in a writer class
- shared parsing or prompt helpers in
resquery/utils/
Current scope:
- multi-turn only
- synchronous only
- prior claims are the main reusable session memory
- follow-up suggestions are user-facing suggestions only, not resolved/unresolved state
dbqueryremains the retrieval and synthesis engine- the next turn query should be contextualized as:
This is what we know so far + prior claims + new user question - claims should be extracted from individual
dbquerybatch summaries, not from the final synthesized summary and not from raw fused chunks - claim evidence should be attached programmatically from the
chunk_idsalready stored on each batch summary
resquery should preserve a compact research session with:
- session metadata
- ordered turns
- prior claims
- follow-up suggestions
- evidence index metadata
resquery should not reintroduce active_focus or resolved/unresolved question tracking unless there is a clear new requirement.
Runtime Model Setup
The repo .venv is for the repo itself and should stay Marker-compatible.
Do not install Gemma 4 support directly into the repo .venv unless there is a strong reason and you understand the dependency impact.
The working Gemma 4 setup is:
- external Python:
/Users/danielparsons/.unsloth/unsloth_gemma4_mlx/bin/python - model identifier:
unsloth/gemma-4-E4B-it-UD-MLX-4bit
The downloaded model weights are not stored in this repo. They live in the Hugging Face cache under the user home directory.
Important operational note:
- inside the Codex sandbox, MLX/Metal may fail to initialize
- outside the sandbox, the Unsloth MLX runtime works correctly on this machine
If you need to prove the real Gemma path works, run it outside the sandbox.
Important Output Semantics
Each classified chunk must preserve:
- original chunk text
- chunk index
- word count
- final label
- source
- confidence
- used_context
- reason
This metadata is downstream pipeline input, not just display output. Preserve alignment with the original chunk objects.
For database rows, also preserve paper-level metadata such as:
paper_idpaper_titleauthorsjournalvolumeissueyeardoiissnreferencespdf_pathmarker_json_pathmarkdown_path
Working Rules
- Prefer filtered Markdown as the source of truth for chunk context.
- Avoid re-chunking unnecessarily if chunk data already exists in the current pipeline step.
- Keep deterministic classification conservative.
- Prefer passing unresolved cases to the LLM rather than adding weak heuristic text rules.
- Never allow unresolved chunks into LanceDB.
- Preserve repo
.venvdependency integrity. - Keep new code modular; avoid putting orchestration, prompting, parsing, and runtime control into one file.
- Treat
Referencesas the end of the paper body. - In
dbquery, keep files short and focused. - In
dbquery, classes should own a single responsibility and should not become long mixed-purpose classes. - In
dbquery, move formatting, ranking math, or text-budget helpers into subfolders such asdbquery/utils/when that keeps primary classes small. - In
dbxquery, keep files short and focused. - In
dbxquery, classes should own a single responsibility and should not become long mixed-purpose classes. - In
dbxquery, keep planner output tightly schema-constrained and validate it before retrieval runs. - In
dbxquery, final answers must be grounded only in retrieved evidence; if evidence is weak or missing, the answer should say so explicitly. - In
resquery, keep files short and focused. - In
resquery, classes should own a single responsibility and should not become long mixed-purpose classes. - In
resquery, prior claims should influence the nextdbqueryturn by contextualizing the query text before retrieval runs. - In
resquery, claim provenance should come from knowndbquerybatch-summarychunk_ids, not from model-generated chunk IDs. - In
resquery, follow-up suggestions are suggestions for the user, not a state machine.
Full Pipeline Behavior
The full runner is resumable by artifact:
- reuse existing Marker JSON unless
--rerun-marker - reuse existing filtered markdown unless
--rerun-filtered-markdown - reuse existing classified chunks unless
--rerun-classification
The full runner writes:
marker/conversion_results/<paper_id>/<paper_id>_marker.logmarker/conversion_results/<paper_id>/<paper_id>_classification.log
Marker output should be streamed to a log file, not captured fully in memory.
During long Marker runs, the pipeline should print elapsed-time progress and allow the user to stop the run without starting parallel jobs.
Validation
When changing classification or filtering code, verify at least:
./.venv/bin/python -m compileall chunker dbinsert- deterministic classification on
marker/conversion_results/2025_Uchida/2025_Uchida_filtered.md - if touching the LLM path, one real fallback chunk through the external Gemma runtime
- if touching filtering, confirm appendix content after references is absent from regenerated filtered markdown
When changing dbquery, dbxquery, or resquery, verify at least:
./.venv/bin/python -m compileall dbquery dbxquery resquery- one
dbqueryrun that writes an output artifact with a synthesized summary - if touching
dbxquery, one real follow-up run through the external Gemma runtime - if touching
resquery, one real multi-turnresquerysession through the external Gemma runtime - if touching
resquery, confirm that:- prior claims are present in the contextualized query text sent to
dbquery - claims are extracted from batch summaries
- claim evidence comes from batch-summary
chunk_ids - follow-up suggestions are written to session state and output artifacts
- prior claims are present in the contextualized query text sent to
- if touching follow-up filtering, confirm the saved trace records the tool calls, retrieved evidence, and grounded final answer
Useful Commands
Deterministic run:
./.venv/bin/python -m chunker.run_section_classification \
marker/conversion_results/2025_Uchida/2025_Uchida_filtered.md
Run with Gemma fallback:
./.venv/bin/python -m chunker.run_section_classification \
marker/conversion_results/2025_Uchida/2025_Uchida_filtered.md \
--model-path unsloth/gemma-4-E4B-it-UD-MLX-4bit \
--python-executable /Users/danielparsons/.unsloth/unsloth_gemma4_mlx/bin/python
Run the full pipeline from a PDF:
./.venv/bin/python -m dbinsert.run_full_pipeline \
pdfs/2025_Zhu.pdf \
--db-path data/lancedb \
--replace-existing
Rerun filtered markdown and classification only:
./.venv/bin/python -m dbinsert.run_full_pipeline \
pdfs/2025_Zhu.pdf \
--db-path data/lancedb \
--replace-existing \
--rerun-filtered-markdown \
--rerun-classification
Run a follow-up query over a prior synthesized summary:
./.venv/bin/python -m dbxquery.run_followup \
"Tell me how Zhu and Uchida differ on this issue?" \
--prior-summary-path outputs/llms_cefr_quality_query.md \
--db-path data/lancedb \
--output-path xoutputs/zhu_vs_uchida_differences_followup.md
Run an iterative resquery session turn:
./.venv/bin/python -m resquery.run_session \
"What are the weaknesses of LLMs?" \
--session-path ressessions/llm_weaknesses.json \
--db-path data/lancedb \
--query-output-path resoutputs/llm_weaknesses_t1_dbquery.md \
--output-path resoutputs/llm_weaknesses_t1_resquery.md
Continue the same resquery session with prior-claim context:
./.venv/bin/python -m resquery.run_session \
"How does chain of thought impact the accuracy of CEFR generated text?" \
--session-path ressessions/llm_weaknesses.json \
--db-path data/lancedb \
--query-output-path resoutputs/llm_weaknesses_t2_dbquery.md \
--output-path resoutputs/llm_weaknesses_t2_resquery.md