This article walks through that pipeline at an architectural level. It's based on the kind of system behind tools like AI Dance Generator, which turns one image into a 9:16 MP4. I'll stay model-agnostic, because the specific checkpoints change every few months while the structure stays the same.

The problem, stated precisely

Given:

• a source image I containing one subject, and
• a driving motion M, which is a sequence of poses p_1 … p_T,

produce a video V = (f_1 … f_T) where each frame shows the subject from I in pose p_t, with:

• identity preserved (face, body shape, clothing),
• temporal coherence (no flicker or texture crawl between frames), and
• plausible occlusion (parts of the body hidden in I must be invented convincingly when they come into view).

That last point is the hard part. A front-facing photo contains no information about the subject's back, so a turn requires the model to hallucinate it consistently.

Stage 1: Input validation and preprocessing

Don't spend GPU time on images that can't work. A cheap CPU or small-GPU pre-pass should:

1. Detect people and reject images with zero or several subjects (or ask the user to choose one).
2. Check coverage. Estimate keypoints and confirm the head and torso are visible. Missing limbs are recoverable; a missing head usually isn't.
3. Normalize. Fix EXIF orientation, convert to RGB, and resize to the model's working resolution while keeping the aspect ratio.
4. Segment the subject. A foreground mask lets you treat the subject and background differently later.

def preprocess(img):
    img = apply_exif_orientation(img).convert("RGB")
    people = detect_people(img)
    if len(people) != 1:
        raise InputError("Please upload a photo with exactly one person.")
    kp = estimate_keypoints(img, people[0])
    if not kp.has(["nose", "left_shoulder", "right_shoulder"]):
        raise InputError("Head and shoulders must be visible.")
    mask = segment_subject(img, people[0])
    return resize_keep_aspect(img, target=768), kp, mask

Returning a clear, specific error here is better UX than returning a broken video later.

Stage 2: Representing the driving motion

The dance is stored as pose data, not as video. Typical representations:

• 2D skeleton keypoints (OpenPose / COCO style): cheap and widely supported, but ambiguous in depth.
• Dense pose / UV maps: map each pixel to a body-surface coordinate, giving richer conditioning.
• 3D body model parameters (SMPL-like): handle rotation and depth best, but cost more to extract.

Motion is extracted once per dance from a reference clip, then cleaned up offline: smooth jitter with a one-euro or Savitzky–Golay filter, fill dropped keypoints, and trim to a clean loop.

Retargeting

The reference dancer and your subject have different proportions. If you feed raw keypoints, a short subject gets stretched limbs. Retarget by:

1. Computing bone lengths for the source subject from its keypoints.
2. Rescaling each bone in the driving sequence to match, while keeping joint angles.
3. Anchoring the root (mid-hip) to the subject's position and scale in the frame.

Skipping this step is one of the most common causes of "something looks off" results.

Stage 3: The generative core

Most current systems use a video diffusion model with two conditioning paths:

• Appearance conditioning: features from the source image, injected via a reference network, cross-attention, or both. This carries identity, clothing, and texture.
• Pose conditioning: the retargeted pose sequence, encoded by a lightweight pose guider and added to the noisy latents, similar in spirit to ControlNet.

Temporal layers (attention across frames) give the model a chance to keep frames consistent with one another. Frames are generated in latent space and decoded by a VAE.

Practical considerations:

• Clip length vs. memory. Attention across frames grows quickly with T. Long videos are usually generated in overlapping windows and blended at the seams.
• Sampling steps. Fewer steps mean faster jobs and more artifacts. Choose per product tier, not globally.
• Seed control. Fix the seed per job so retries are reproducible when you debug.

Stage 4: Temporal consistency and cleanup

Even good models produce flicker in high-frequency areas such as hair, patterned clothing, and hands. Mitigations, from cheapest to most expensive:

• Overlap-and-blend windows with a crossfade over the shared frames.
• Background locking: composite the generated subject over a stabilized version of the original background using the mask from Stage 1, so the background doesn't "breathe."
• Face restoration on a per-frame crop, blended back with a feathered mask. Use it lightly, because overdoing it creates a pasted-on look.
• Frame interpolation to raise the frame rate after generating at a lower one.

Stage 5: Framing for 9:16

The output targets vertical platforms, so composition matters:

• Track the subject's bounding box across all frames, then compute one smoothed crop path. Per-frame cropping causes a jittery virtual camera.
• Leave headroom. Platform UI covers the top and bottom of the screen.
• Pad rather than crop when the dance moves wide, so arms aren't cut off.

Stage 6: Encoding and delivery

Encode for maximum compatibility:

ffmpeg -framerate 24 -i frames/%05d.png \
  -vf "scale=1080:1920:flags=lanczos,format=yuv420p" \
  -c:v libx264 -profile:v high -crf 18 -preset slow \
  -movflags +faststart output.mp4

• yuv420p and H.264 High play everywhere.
• +faststart moves the metadata to the front so the video starts playing before it fully downloads.
• Add a silent audio track if a platform rejects video-only files, or mux in the dance's music where licensing allows.

System architecture

Generation takes seconds to minutes, so it must be asynchronous:

1. API receives the upload and runs Stage 1 synchronously, so bad input fails fast.
2. A job goes into a queue (Redis, SQS, or similar) with an idempotency key, so a double-click doesn't bill or render twice.
3. GPU workers pull jobs, run Stages 2–6, and upload the MP4 to object storage.
4. The client polls or subscribes to a status endpoint, then gets a signed URL.

This structure also fits per-output billing, which is what a pay-per-video product does: you charge when a job is created and finished, which lines up neatly with GPU cost per job.

Testing a generative pipeline

Unit tests won't tell you whether a dance looks good, but you can still automate a lot:

• Golden inputs: a fixed set of tricky photos (side profile, loose clothing, pet, low light) rendered on every model or config change.
• Cheap metrics: face-embedding similarity between source and frames (identity drift), and frame-to-frame difference in background regions (flicker).
• Human review of a contact sheet before shipping a new checkpoint.

Takeaways

• Most visible quality problems come from preprocessing, retargeting, and framing, not the diffusion model itself.
• Treat motion as clean, retargeted pose data prepared offline.
• Design for asynchronous, idempotent jobs from the start.
• Encode boring, compatible MP4s: 1080×1920, H.264, yuv420p, faststart.

If you want to see what this pipeline produces for a non-technical user, the make a photo dance flow is a compact example of hiding all six stages behind a single upload button.

Loretypes is a small site with six self-discovery quizzes: Aura Color, Archetype, Moral Alignment, Past Life, Color Season, and Spirit Animal. They're meant for self-reflection and fun, not diagnosis. Past Life is openly fictional storytelling, and Color Season asks which color combinations you enjoy rather than looking at your face or skin.

From an engineering point of view, the interesting part is that the six quizzes share most of their plumbing (routing, submission, storage, share links, images, SEO) while their scoring rules and result screens differ. The Archetype quiz scores bipolar dimensions; the Aura quiz sums answers across seven energy dimensions and maps them to one of eight colors, with White as the result when they're balanced. This post walks through how the codebase holds those together, and where the shared abstraction deliberately stops.

One registry, many routes

Everything starts in lib/tests/registry.ts:

export const testRegistry = [
  auraColorDefinition,
  archetypeDefinition,
  moralAlignmentDefinition,
  pastLifeDefinition,
  colorSeasonDefinition,
  spiritAnimalDefinition,
] as const satisfies readonly TestDefinition[];

validateRegistry([...testRegistry]);

export function getTestByCluster(cluster: string): TestDefinition | undefined {
  return testRegistry.find((definition) => definition.cluster === cluster);
}

Each definition lives in lib/tests/definitions/ and implements the TestDefinition interface from lib/tests/types.ts: questions, answer scale, scoring configuration, result types, SEO copy, theme tokens, a visualization id, an OG template id, a version, and a key for saving drafts in the browser. A quiz has two identifiers: a slug used by the API and storage, and a cluster used in URLs. They can differ: Moral Alignment's slug is moral-alignment, while its URL is /alignment/.

validateRegistry runs when the module loads, so a bad definition fails early. It rejects reserved route segments, duplicate slugs, and route segments that collide across quizzes.

The App Router pages are thin consumers of that registry. The quiz landing page at app/[cluster]/page.tsx enumerates its routes straight from it:

export function generateStaticParams() {
  return testRegistry.map((definition) => ({ cluster: definition.cluster }));
}

export const dynamicParams = false;

The same definition supplies that page's metadata and its FAQ structured data. The saved-result page uses it to resolve result copy, and the image route uses its OG template id to pick a renderer.

That said, this isn't "drop in a config file and get a new quiz." Definitions contain functions (for example mapVisualizationScores), and the landing page explicitly branches into a specialized React experience per quiz. The registry connects content and infrastructure; it doesn't remove quiz-specific UI work.

One runner, scoring on the server

All six quiz experiences render the same QuizRunner component (components/quiz/quiz-runner.tsx). When you answer the last question, it posts only the answer array:

const response = await fetch(`/api/tests/${definition.slug}/results/`, {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({ answers: finalAnswers }),
  signal: controller.signal,
});

The handler in app/api/tests/[slug]/results/route.ts caps the body at 4 KB, applies a rate limit to result creation, looks up the definition by slug, and then calls processSubmission. The browser never tells the server which result it got. validateSubmission checks that answers is an array of exactly questionCount integers within the quiz's allowed range, and the server computes the result itself.

Scoring dispatch is a small strategy table in lib/scoring/engine.ts:

export function scoreAnswers(definition: TestDefinition, answers: number[]): ResultPayload {
  const runner = strategyRegistry[definition.scoring.type];
  if (!runner) {
    throw new Error(`Scoring strategy is not implemented: ${definition.scoring.type}`);
  }
  return runner(definition, answers);
}

Four strategies are implemented: dimensional sum, bipolar dimensions, an alignment axis grid, and a seasonal profile. The strategy table is typed as a Partial record, which is honest: one strategy name in the type union (weighted-type) has no runner yet, and the engine throws instead of guessing.

The alignment strategy shows how simple the arithmetic can be. In lib/scoring/strategies/axis-grid.ts, each 1–5 answer is centered on 3 and multiplied by the question's direction:

// Simplified: the question-kind check is omitted.
definition.questions.forEach((question, index) => {
  const contribution = (answers[index] - 3) * question.direction;
  if (question.axis === "lawChaos") lawChaos += contribution;
  else goodEvil += contribution;
});
const cell = cellFromAxes(lawChaos, goodEvil);

Dimensional scoring works differently. It sums answers per dimension, breaks ties by a configured order, and can return a "balanced" result when the spread between dimensions is small. The quizzes share an endpoint, not a formula.

A share link points to a stored, versioned result

After scoring, saveResultWithRetry in lib/results.ts stores a record with a 12-character lowercase alphanumeric share id (nanoid with a custom alphabet), plus the test slug, test version, scored payload, creation time, and an expiry that new records set to null. The raw answer array isn't part of that record.

On an id collision, the save loop tries a fresh id, up to five attempts, and rethrows any other storage error. The store, id factory, and clock are all injectable, which matters for testing (more below).

Storing the version is only useful if reads respect it. lib/tests/result-version.ts maps a stored record back to the definition it was scored with. Archetype keeps an explicit archived 0.2.0 definition, so old results keep rendering with the copy and types they were created with. A small read-only adapter also translates an older payload shape into the current field names, without rewriting the stored data:

// Explicit archive, never a fallback to whichever definition is current.
const archetypeVersions: Record<string, TestDefinition> = { "0.2.0": v020 };

Not every quiz has a frozen archive yet. Alignment, Past Life, and Color Season have explicit version maps, Spirit Animal accepts only its current version, and Aura falls through to the current definition. The point is that each quiz has an explicit read policy, rather than a version field nobody checks.

The result page, app/[cluster]/r/[shareId]/page.tsx, validates the id format, loads the record, resolves the definition for its version, and picks the right result component. An unknown id or incompatible version ends in a 404.

Static pages, dynamic results, and metadata

Content routes are statically generated. The landing page, the detail index and type pages, and the learn articles all export generateStaticParams derived from the registry, with dynamicParams = false, so unknown slugs 404 instead of rendering on demand. The parameter lists aren't a blind cross product. A definition can turn detail pages off with detailPages: false, and Past Life currently has no type pages or learn articles.

Saved results are the opposite: there can be any number of them, so the result page looks records up by id rather than enumerating them. The result image and poster routes declare runtime = "nodejs" and dynamic = "force-dynamic", and so does the sitemap.

Metadata is centralized in lib/seo/metadata.ts, which builds titles, descriptions, canonical URLs, Open Graph, and Twitter cards. Landing pages and articles point at pre-rendered static images. Result pages get a per-result image and are deliberately kept out of the index:

return {
  title,
  description,
  alternates: { canonical },
  robots: { index: false, follow: true },
  openGraph: { title, description, url: canonical, images: [image] },
  twitter: { card: "summary_large_image", title, description, images: [image] },
};

The sitemap is built from indexableClusterPaths, which leaves out individual results and filters Archetype type pages through a content-quality gate. robots.ts allows crawling and points to the sitemap. Indexing is then controlled per page: crawlers can follow a result link, but results don't compete with the content pages in search.

Share images are rendered on demand

Each result has an OG image at /[cluster]/og/[shareId].png. The route delegates to resultImageResponse in lib/result-image-route.ts, which validates the id, loads the record, resolves its versioned definition, and picks a renderer from lib/result-image/index.ts using the definition's OG template id. The renderers use ImageResponse from next/og.

A successful render returns a PNG with Cache-Control: public, max-age=31536000, immutable, which is safe because a stored result is never updated (the store interface only has create and find). A render failure returns a 500 with no-store, so a broken image doesn't get cached for a year. Nothing is written to storage; the PNG is regenerated from the saved payload when a cache misses it.

Posters are a separate, opt-in capability. Past Life has no poster. Aura renders a 1200×630 OG image, a 1080×1080 square poster, and a 1080×1920 story poster. Archetype and Alignment currently use one square layout for both poster routes.

Share buttons build their URLs with a tiny helper in lib/share-url.ts, which strips any query or hash and adds a channel marker:

export function buildShareUrl(origin: string, pathname: string, channel: ShareChannel) {
  const cleanPathname = pathname.split(/[?#]/, 1)[0];
  const url = new URL(cleanPathname, origin);
  url.searchParams.set("s", channel);
  return url.toString();
}

612 tests, mostly about contracts

The suite runs on Vitest. At the time of writing it's 72 test files and 612 tests. vitest.config.mts defaults to the Node environment. The files that need a DOM opt in with a // @vitest-environment jsdom comment at the top.

A few patterns do most of the work:

• Registry tests build deliberately invalid definitions (reserved segments, duplicate slugs, colliding routes, inconsistent SEO) and assert that validateRegistry throws.
• Persistence tests inject a fake store that throws duplicate-id errors, to check both "succeeds on a later attempt" and "gives up after five."
• Server components rendered to strings. About a dozen test files await an async server component directly, pass the tree to renderToStaticMarkup, and assert on headings, links, accessible table markup, and JSON-LD, without a browser.
• Golden HTML fixtures in tests/__fixtures__/. The homepage test renders <HomePage /> and compares the result with toBe against home-portal.html, byte for byte. The aura learn-article tests compare against their fixtures after stripping the shared footer, and separately assert one specific cross-link paragraph.

Two lessons

A version field needs a read policy. Writing testVersion into every record is easy. The hard part is deciding what happens when an old record is read after the copy or result types change. For Archetype, that meant keeping an explicit archive of the old definition and a read-only adapter for an earlier payload shape. Making the policy explicit per quiz (archive, exact match, or "current is fine") has been more useful than pretending every quiz is equally frozen.

Byte-for-byte fixtures are noisy on purpose. The homepage fixture fails on any markup change, including a new footer link. That sounds annoying, but it turns "did this change anything else on the page?" into a diff you review. When the change is intended, you regenerate the fixture in the same commit and the diff shows exactly what moved. For pages where only part of the markup matters, like the learn articles, stripping the shared footer first keeps the fixture focused on the content it's meant to protect.

I used AI assistance to help draft this post; the code and project are my own.