One pipeline, four creative formats, one product identity.
Product images and a prompt in. Image bundles, fast paced videos, scripted videos, and UGC style cuts out. The challenge was supporting very different creative formats without turning the backend into a set of tightly coupled provider scripts.
- Role
- Engineer · routing + pipeline
- Team
- Gen AI service
- Stack
- FastAPI, Python, Postgres, RabbitMQ
- Surface
- Internal job APIs, callbacks
- Formats
- 4 output modes
- Sanitized
- Yes
From a product image and a prompt to a full campaign surface.
The workflow generates product focused creative from product images, optional links, user prompts, and campaign settings. I structured the system around strict job contracts, durable step state, capability based provider adapters, and branch specific pipelines that share common reliability primitives.
- Product creation
- Metadata extraction
- Storyboard suggestions
- Image bundles
- Fast paced videos
- Script to video
- UGC style outputs
- Product fidelity checks
A wide creative surface with different pipeline shapes.
A single product can require clean imagery, metadata, storyboard concepts, scene images, video clips, scripts, voiceover, continuity frames, and final stitched media. Different output modes have different shapes, but they still need consistent progress tracking, retries, and completion callbacks.
Provider scripts wired into product code.
- · Format changes touched product APIs.
- · New providers required broad refactors.
- · Retries reran whole campaigns.
- · Fidelity checks lived outside the pipeline.
One orchestration layer, branch aware pipelines.
- · Adding a format is a routing concern.
- · Adapters keep providers swappable.
- · Retries resume from scene or stage.
- · Fidelity is a pipeline stage, not a review step.
Normalized job contracts
For marketing creation, output generation, and retry jobs.
Branch aware routing
Image bundle, video, script to video, and UGC style flows behind one entry point.
Persisted job & step state
For observability, audit, and targeted retries.
Capability adapters
Image, video, audio, script, storyboard, fidelity, stitch, lip sync.
Lifecycle events
Structured accepted, started, progress, completed, and failed events.
Contract & pipeline tests
Routing, callbacks, persistence, output shape.
Adding a new output mode is a routing concern, not a platform change.
Internal requests are validated, persisted with idempotency, and dispatched onto dedicated queues for asynchronous execution. The product backend submits a compact product snapshot. The Gen AI service handles media execution, intermediate assets, failure recovery, and completion reporting.
Product backend
- Users & workspaces
- Permissions
- Public API
- Product records
- Response shapes
Gen AI service
- Job APIs (idempotent)
- Persistence + snapshots
- Worker queues
- Output mode router
- Provider adapters
Identity separated from campaign specific instructions.
Later creative generation reuses a stable product representation while still varying format, pacing, scene style, and presentation mode. The service records stage outputs as reusable checkpoints rather than treating each generation attempt as disposable.
- 01Neutral imageClean product image from user media.
- 02MetadataProduct attributes and category.
- 03Character sheetReference for visual consistency.
- 04Link metadataOptional product page extraction.
- 05Storyboard seedsConcepts that seed later output.
- 06EmitProgress and completion to product record.
One entry point, four branch specific pipelines.
The output router inspects the requested format and dispatches to a branch specific pipeline while preserving common job lifecycle, progress events, retry semantics, and final output structure.
Image bundle
01Storyboard concepts, scene image generation, then bundle finalization.
Fast paced video
02Cut storyboard, scene images and videos, then stitching.
Script to video
03Script interpretation or segmentation, spoken media, then timed video.
UGC style
04Prompt interpretation, spoken script and storyboard, media, final assembly.
Fidelity is a pipeline stage, not a review step.
Outputs must remain visually faithful to the source product while still allowing creative variation. The pipeline separates product identity from output specific creative instructions and runs fidelity checks with controlled regeneration when assets drift.
- product only
- with model
- mixed scenes
Polymorphic job contracts
Creation, output generation, and retries share contracts while keeping payload validation strict.
Output mode routing
Image bundle, fast paced video, script to video, and UGC under shared lifecycle semantics.
Intermediate asset tracking
Scene images, videos, scripts, audio, transcripts, and storyboard metadata reported incrementally.
Product fidelity loop
Output generation validates consistency and performs controlled regeneration when assets drift.
Scene aware retries
Image and video stages can resume from a specific scene order without discarding usable work.
Provider isolation
Orchestration depends on media capabilities, not account or vendor specific details.
Retry context carries stage and scene order.
Useful for operational recovery and for creative iteration. Avoids full pipeline reruns when only one scene or stage needs regeneration.
- 01Neutral product imageRerun cleanup without touching metadata or storyboard.
- 02Storyboard suggestionsRegenerate while preserving extracted product metadata.
- 03Image from sceneRetry image generation from a specific scene order.
- 04Video from sceneRetry video generation from a specific scene order.
- 05Resume from stagePick up an output pipeline at a named stage.
Idempotent submission
Duplicate internal requests resolve predictably.
Durable job steps
Every major stage records status, inputs, outputs, errors.
Callback outbox
Outbound events persisted for delivery tracking and retry.
Queue isolation
Marketing workers scale separately from avatar and callback.
Provider adapters
Orchestration independent from provider request details.
Snapshot based jobs
Execution from captured product state, not product DB models.
Failure localization
Retries start at point of failure, not whole campaign.
Private internal APIs and service to service authentication.
Secrets, provider credentials, storage configuration, callback destinations, and account specific values come from environment configuration and are not included here. Job payloads carry only the product and output snapshots needed for generation. The service does not require access to product user credentials or the product database.
A cleaner platform for product creative generation.
AI execution was separated from the product backend. The Gen AI service now owns generation orchestration, retries, provider integration, progress state, and event delivery, while the core backend stays focused on product records, permissions, and public APIs. Extensible by output mode, observable by stage, resilient to provider failures, and structured for fast creative iteration.