Imported from PKU-YuanGroup/OpenAI4S (skills/bioskills/bio-chemoinformatics-generative-design/SKILL.md). Install upstream with npx skills add PKU-YuanGroup/OpenAI4S --skill bio-chemoinformatics-generative-design. Copyright stays with the author.

Version Compatibility

Reference examples tested with: REINVENT 4.0+, RDKit 2024.09+, PyTorch 2.1+, MolMIM (NVIDIA BioNeMo), chemprop 2.0+.

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Generative Molecular Design

Generate novel molecules biased toward desired properties using deep generative models. REINVENT 4 (Loeffler et al. 2024, AstraZeneca) provides four generator families: Reinvent (de novo), Libinvent (scaffold decoration and library design), Linkinvent (linker design), and Mol2Mol (similarity-constrained molecular optimization). These support design tasks including R-group replacement and scaffold hopping and can be used with transfer learning, reinforcement learning, and curriculum learning. For specific niches: MolMIM (NVIDIA BioNeMo) for latent-space property optimization, DiffSMol / DiGress for diffusion-based generation, and JT-VAE for latent-space optimization. The art of generative design is in the scoring function: poorly-designed scoring rewards uninteresting molecules, while well-designed scoring captures both activity and developability.

For QSAR/scoring models that feed generative design, see chemoinformatics/qsar-modeling. For synthetic feasibility, see chemoinformatics/retrosynthesis. For library enumeration as alternative, see chemoinformatics/reaction-enumeration.

Generator Mode Taxonomy

Learning Algorithm Taxonomy

Decision Tree by Scenario

REINVENT 4 Setup

REINVENT 4 uses a TOML configuration file specifying generator, algorithm, prior model, and scoring functions.

Goal: Configure a reinforcement-learning REINVENT 4 run with a prior, agent, sampling parameters, and a QED scoring component.

Approach: Build a release-matched REINVENT 4 staged-learning TOML config with [parameters] for the prior/agent checkpoints, [[stage]] blocks, and one or more [[stage.scoring.component]] blocks. Validate the config with the installed release because component parameters evolve between versions.

run_type = "staged_learning"
device = "cuda:0"

[parameters]
prior_file = "priors/reinvent.prior"
agent_file = "priors/reinvent.prior"
batch_size = 64
unique_sequences = true

[[stage]]
termination = "simple"
min_steps = 25
max_steps = 500

[stage.scoring]
type = "geometric_mean"

[[stage.scoring.component]]
[stage.scoring.component.QED]

[[stage.scoring.component.QED.endpoint]]
name = "QED"
weight = 1

# The REINVENT 4 CLI binary is `reinvent` (not `reinvent4`).
reinvent -l logfile.log config.toml

Output: a live stage CSV using summary_csv_prefix, plus the configured chkpt_file at stage termination or graceful interruption. Post-process the CSV to select molecules; REINVENT does not emit a checkpoint and SMILES file at every iteration by default.

Scoring Function Design (Most Important Part)

A good scoring function:

• Returns 0-1 (normalized)
• Combines multiple endpoints
• Penalizes pathological generations (PAINS, unstable, unsynthesizable)

Goal: Build a multi-component generative reward that balances predicted activity, drug-likeness, synthesizability, and novelty.

Approach: Combine a QSAR sigmoid on pIC50, QED, SA-score reverse-sigmoid, and Tanimoto-similarity reverse-sigmoid via geometric mean so any zero component zeroes the total.

In REINVENT 4, define these under the active stage's [stage.scoring] section, with each component using the exact component and endpoint tables from the installed release's configuration examples. Do not reuse REINVENT 3 [scoring_function] or [[scoring_function.components]] syntax in a REINVENT 4 config. The accompanying example is deliberately limited to built-in, documented component structure; add predictive-property endpoints only after validating their release-specific model-container parameters.

geometric_mean ensures all components must be reasonably high (one zero -> zero total). arithmetic_mean allows compensation.

Multi-Parameter Optimization (MPO)

Lead optimization commonly involves multiple objectives. The following component types illustrate a project-specific scoring design; weights and transforms must be fit to the actual assays and decision context.

The weights and transformation bounds above are repository starting examples only. Normalize weights as required by the selected aggregation and tune every bound against project assay distributions and prospective behavior.

Reward Hacking (Production Pitfall)

RL agents will find ways to maximize reward without learning the intended behavior:

• Trivial scaffolds that score high on QED
• Repeat structural motifs that game similarity scoring
• Out-of-distribution molecules that exploit QSAR overconfidence
• Trivial SMILES (e.g., "CCC...C") that match generic scoring

Mitigations:

• Include one or more synthesis-aware signals when they have been validated for the project; SA score alone does not establish route feasibility
• Use ensemble QSAR with uncertainty (penalize high-uncertainty predictions)
• Include diversity bonus (Tanimoto to reference)
• Add fingerprint similarity penalty within batch (prevent mode collapse)
• Validate generated samples on held-out QSAR test set

Synthetic Accessibility Scoring

sa_score (Ertl 2009) measures synthetic accessibility: 1 (easy) to 10 (very hard).

from rdkit.Contrib.SA_Score import sascorer
from rdkit import Chem

def sa_score(smi):
    mol = Chem.MolFromSmiles(smi)
    if mol is None:
        return None
    return sascorer.calculateScore(mol)

sascorer is shipped in RDKit Contrib in current RDKit distributions. Use the namespaced import above; do not install an unrelated top-level package.

The score is a 1-to-10 heuristic derived from fragment contributions and molecular complexity, with lower values intended to indicate easier synthesis. It is not a route planner, cost estimate, or calibrated feasibility probability. Use it as one audited reward component or annotation, never as an absolute filter.

Diffusion-Based Generation (Modern Alternatives)

Diffusion models iteratively denoise molecular representations, whereas REINVENT generators autoregressively construct SMILES. Diversity, validity, and drug-likeness depend on the model, training data, conditioning, and evaluation protocol; compare them on a matched benchmark for the intended task.

Constrained / Goal-Directed Generation

Goal: Enforce hard structural requirements (e.g., must contain hydroxyl) and exclude PAINS without letting constraint satisfaction game the reward.

Approach: Stage transfer learning then RL. In REINVENT 4, CustomAlerts is a global structural-alert filter: a match produces zero and it is applied before score aggregation. MatchingSubstructure is a scoring component (1 for a match and 0.5 otherwise), so it is a soft penalty rather than a hard inclusion constraint. Apply a separate post-generation SMARTS validation step when presence of a feature is mandatory.

[[stage.scoring.component]]
[stage.scoring.component.CustomAlerts]

[[stage.scoring.component.CustomAlerts.endpoint]]
name = "Unwanted SMARTS"
params.smarts = ["PAINS_SMARTS_1", "BRENK_SMARTS_1"]

[[stage.scoring.component]]
[stage.scoring.component.MatchingSubstructure]

[[stage.scoring.component.MatchingSubstructure.endpoint]]
name = "Hydroxyl preference"
weight = 0.1
params.smarts = ["[OX2H]"]

There is no REINVENT 4 filter_only option for these components. Treat structural alerts as triage flags where appropriate, and separately verify any true hard inclusion or exclusion rule on the generated structures.

MolMIM (NVIDIA BioNeMo)

MolMIM encodes SMILES into a learned latent space, uses gradient-free CMA-ES to optimize a user-defined property objective, and decodes candidate molecules.

# Pseudo-code; requires NVIDIA NIM access
# from bionemo.molmim import MolMIMOptimizer
# optimizer = MolMIMOptimizer(model="molmim-property-optimizer")
# optimized = optimizer.optimize(seed_smiles, target_property="logp", target_value=2.0)

Tradeoffs against REINVENT depend on the oracle budget, objective, and implementation; benchmark both under matched constraints when selecting a generator.

Per-Tool Failure Modes

REINVENT RL -- mode collapse

Trigger: The reward, learning strategy, or diversity control favors a narrow chemotype.

Mechanism: Agent finds a high-scoring local maximum and stops exploring.

Symptom: Diversity and scaffold coverage collapse relative to a project-defined baseline while reward continues to rise.

Fix: Add diversity bonus to scoring; reduce sigma; reset agent if collapsed.

REINVENT TL -- overfitting

Trigger: Transfer learning data are too small or homogeneous for the intended generalization task.

Mechanism: Generator memorizes training set; no generalization.

Symptom: Generated molecules near-identical to training set actives.

Fix: Use larger training set; mix with diverse external sample; apply RL after TL.

Generated molecule unsynthesizable

Trigger: SA score missing from reward.

Mechanism: The reward omits synthesis evidence, allowing candidates with no plausible validated route to score well.

Symptom: AiZynthFinder cannot solve route; medchem rejects.

Fix: Combine audited synthesis-aware annotations with reaction- or route-based validation for selected candidates.

PAINS in generation

Trigger: No structural alerts in scoring.

Mechanism: Curcumin / rhodanine / quinone scaffolds optimize for activity (false positives in training data).

Symptom: Generated molecules match PAINS_A.

Fix: Flag relevant structural alerts for orthogonal assay review or apply a project-justified penalty; never reward a PAINS match.

Diffusion model OOD

Trigger: Pocket-conditioned diffusion on novel target family.

Mechanism: Training distribution covered specific protein families; novel targets extrapolate.

Symptom: Generated molecules look like training distribution, not optimized for target.

Fix: Validate on target-family-held-out evaluation; supplement with classical methods.

Validation set leakage

Trigger: Same molecules in training generators and downstream QSAR.

Mechanism: Scoring model has seen the molecule; predictions optimistic.

Symptom: Held-out QSAR validation fails on top generated.

Fix: Use scaffold-split QSAR; ensure scoring model trained on a held-out set vs generation samples.

Reconciliation: REINVENT vs Diffusion

Common Errors

References

• Loeffler et al., J. Cheminformatics 16:20 (2024) -- REINVENT 4 framework and four generator families (DOI 10.1186/s13321-024-00812-5).
• Olivecrona M et al., J. Cheminformatics 9:48 (2017) -- REINVENT original (DOI 10.1186/s13321-017-0235-x).
• Vignac et al., ICLR (2023) -- DiGress discrete diffusion.
• Chen H et al., Nat. Mach. Intell. 7:758-770 (2025) -- DiffSMol structure-based 3D molecular generation (DOI 10.1038/s42256-025-01030-w).
• Peng X, Guan J, Liu Q, Ma J. Proc. ICML, PMLR 202:27611-27629 (2023) -- MolDiff full-atom molecular diffusion.
• Guan J et al., ICLR (2023) -- TargetDiff pocket-conditioned 3D equivariant diffusion (OpenReview: kJqXEPXMsE0).
• Reidenbach D, Livne M, Ilango RK, Gill M, Israeli J. MLDD Workshop at ICLR (2023) -- MolMIM and CMA-ES latent-space optimization (OpenReview: iOJlwUTUyrN).
• Jin W, Barzilay R, Jaakkola T. Proc. ICML, PMLR 80:2323-2332 (2018) -- JT-VAE junction-tree.
• Ertl P, Schuffenhauer A. J. Cheminformatics 1:8 (2009) -- SA score (DOI 10.1186/1758-2946-1-8).
• REINVENT 4 official repository and release-matched configs: https://github.com/MolecularAI/REINVENT4
• RDKit SA-score implementation: https://github.com/rdkit/rdkit/tree/master/Contrib/SA_Score

Related Skills

• chemoinformatics/qsar-modeling - Build scoring models for generative
• chemoinformatics/retrosynthesis - Validate synthetic feasibility post-generation
• chemoinformatics/molecular-standardization - Standardize generated SMILES
• chemoinformatics/admet-prediction - ADMET in scoring components
• chemoinformatics/substructure-search - PAINS / BRENK filter for generation
• chemoinformatics/scaffold-analysis - Scaffold-aware generation control
• chemoinformatics/reaction-enumeration - Alternative to generative for combinatorial
• chemoinformatics/virtual-screening - Validate generated against target