Imported from happyvertical/smrt (packages/video/AGENTS.md). Install upstream with npx skills add happyvertical/smrt --skill video. Copyright stays with the author.

@happyvertical/smrt-video

AI video production pipeline: characters, performers, scenes, shots, sequences, compositions, and ComfyUI workflow integration.

Models

• Character (renamed from PersonalityProfile): visual + voice identity. imageAssetId, voiceProfileId FK, branding config (logo, colors, fonts, lower-thirds). Scene-specific position/scale configs.
• Performer: physical likeness for IP-Adapter FaceID. faceEmbedding (512-dim), referenceAssetIds, ipAdapterWeight.
• Scene: virtual background. sourceType (image/video/panorama_360/panorama_180), projection, viewpoints array (pan/tilt/fov), lightingProfile, anchorPoints.
• VideoShot (extends Content): atomic generation unit. durationInFrames, scriptText, scriptWordCount. Estimated duration: scriptWordCount / 2.7 words/sec (±15% tolerance). videoMetadata includes wordTimings for lip-sync.
• VideoSequence (extends Content): thematic section with transitionType (none/fade/slide/wipe), position ordering.
• VideoComposition (extends Content): top-level container — fps, width, height, durationInFrames, renderStatus/renderProgress.
• VideoShotCharacter: junction (shot ↔ character) with role (primary/secondary/background), position, duration.
• VideoWorkflow: ComfyUI template. workflowJson, nodeMapping (maps semantic names → node IDs), workflowType (prebake/broadcast/lipsync/postprod/custom). injectParameters() deep-clones workflow + overwrites node.inputs.

Design Principles

• Store frames, compute seconds: durationInFrames everywhere; seconds = frames / fps
• Content inheritance: VideoShot, VideoSequence, VideoComposition all extend Content from smrt-content
• Hierarchy: Composition → Sequence → Shot → ShotCharacter
• Owned asset normalization: CharacterOwnedAsset, PerformerOwnedAsset, and SceneOwnedAsset are canonical noun joins; VideoShot/VideoSequence/VideoComposition use inherited content_assets

Gotchas

• PersonalityProfile → Character rename: old name still exported for backwards compatibility
• Legacy video STI asset rows: reads still union canonical joins with legacy STI asset records during migration, but new writes go through the model helpers / join tables
• Schema rollout: run smrt db:migrate before relying on character_assets / performer_assets / scene_assets writes; reads still tolerate those tables being absent during migration
• ComfyUI injectParameters(): assumes node.inputs exists; warnings logged in dev mode
• Frame arithmetic: trimBeforeFrames/trimAfterFrames require manual effective frame calculation
• Word timings from external TTS: wordTimings array not generated by framework — requires external provider
• IP-Adapter weights: faceEmbedding upload not handled; weight is metadata-only (0.5-1.0)
• Optional tenancy on all core models