Character AI API (v1)

This API enables clients to turn a single "reference" image of a subject into a full video sequence in which that subject appears consistently—regardless of changes in pose, background, or action. By uploading one image, consumers can programmatically request new frames or entire videos that preserve the subject's visual identity, driven by arbitrary control signals (e.g. poses, text prompts, or latent codes).

Key goals:

• Simplicity: One reference image → many generated frames.
• Flexibility: Support diverse control modalities (pose, text, custom latents).
• Consistency: Enforce identity preservation and temporal smoothness.
• Scalability: Offer both single-frame and batch-oriented video endpoints, with async/webhook support.

1. Upload & Process Reference Image

   • Client uploads I_ref.
   • API returns a reference_image_id.
   • Segmentation and feature-extraction run to produce a mask M_ref and embedding f_identity.

2. Generate Control Sequence

   • Client requests control vectors via pose-, text-, or latent-based endpoints.
   • API returns one or more z_t vectors describing desired per-frame variation.

3. Frame / Video Generation

   • Single-frame: supply one z_t → get I_t.
   • Multi-frame: supply [z_1…z_T] → get a packaged video (MP4/GIF).

4. Monitor & Retrieve

   • Synchronous responses for small jobs.
   • Asynchronous jobs with job IDs, polling, or webhooks for larger video renders.

All endpoints are hosted under the versioned base URL:

https://api.aicharacter.studio/v1

Note: All requests must use HTTPS.

API Key Management

• Generate and manage keys in your dashboard at aicharacter.studio.

• Keys are tied to scopes; each key bears a set of permissions:

  • reference:read – upload and fetch reference-image data
  • control:write – generate control vectors (pose/text/latent)
  • generate:write – produce frames or videos

Sample Request

POST /v1/reference-image HTTP/1.1
Host: api.aicharacter.studio
Authorization: Bearer sk_live_abc123…
Content-Type: multipart/form-data; boundary=----XYZ

------XYZ
Content-Disposition: form-data; name="image"; filename="subject.jpg"
Content-Type: image/jpeg

<…binary image data…>
------XYZ--

When you approach or exceed your limit, the API will respond with HTTP 429 and include:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1618881600

• Retry-After: seconds until you may retry.
• X-RateLimit-Reset: UNIX timestamp when the window resets.

All errors return a JSON payload in the following format:

{
  "error": {
    "code": "string",
    "message": "string",
    "details": { … }
  }
}

Example: Invalid JSON

HTTP/1.1 400 Bad Request
Content-Type: application/json

{
  "error": {
    "code": "BadRequest",
    "message": "Request body could not be parsed as JSON.",
    "details": {
      "line": 4,
      "column": 15
    }
  }
}

This section covers endpoints for uploading a reference image, retrieving its metadata, obtaining a segmentation mask, and extracting the identity feature vector.

Endpoints in this section generate per-frame control vectors (zₜ) that drive pose, text, or custom latent–based variation. Clients supply the reference_image_id and modality‐specific inputs; the API returns one or more zₜ vectors for downstream frame/video generation.

Endpoints in this section consume one or more control vectors (zₜ) along with the reference_image_id to render either a single frame or a full video.

This section describes two higher-level capabilities: 3D/generalization via NeRF and batch-style multi-reference processing.

8.1 3D Pose & View Generalization

Integration with NeRF (Neural Radiance Fields) for enhanced 3D pose and view generalization. Specific endpoint for direct NeRF interaction might be part of specialized workflows or future extensions. Current capabilities are implicitly leveraged in models supporting 3D awareness.

8.2 Batch Processing

Functionality for processing multiple requests efficiently.

To meet the needs of diverse workloads—from single-frame tests to large-scale video batches—our API offers both synchronous and asynchronous modes, fine-grained job management, and options for GPU sizing, caching, and CDN delivery.

9.1 Asynchronous Jobs & Webhooks

• async Flag Add "async": true in your /generate/video or /v1/batch/video requests to enqueue work without blocking your connection.

• Job Creation Response

  HTTP/1.1 202 Accepted
  Location: /v1/jobs/{job_id}
  {
    "job_id": "job_abc123",
    "status_url": "https://api.aicharacter.studio/v1/jobs/job_abc123"
  }
  

• Callback/Webhook Supply "callback_url": "https://your.app/webhook" to have job results POSTed when complete:

  {
    "job_id": "job_abc123",
    "status": "completed",
    "result": { /* identical to sync response */ }
  }
  

9.2 Progress Polling

• Endpoint

  GET /v1/jobs/{job_id}
  

• Response Fields

  Field	Type	Description
  status	string	pending, processing, completed, failed
  progress	number	Percentage 0–100 of completion
  result	object	Populated when status=completed
  error	object	Populated when status=failed

• Example

  GET /v1/jobs/job_abc123 HTTP/1.1
  Authorization: Bearer sk_live_…
  

  {
    "job_id": "job_abc123",
    "status": "processing",
    "progress": 47.5
  }
  

9.3 GPU & Instance Sizing

By default, jobs run on a shared GPU cluster. For predictable performance:

• X-GPU-Type Header (optional):

  • v100 (default)
  • a100
  • t4

• X-Instance-Count Header (async only):

  • Number of GPU workers (1–4)

Note: Additional charges apply for dedicated hardware.

9.4 Caching & Deduplication

• ETags & Conditional Requests

  • Frame/video endpoints return an ETag header.
  • Clients may GET with If-None-Match to reuse cached results when control inputs and reference image are unchanged. (Note: For POST generation endpoints, this implies re-POSTing with same parameters might yield cached result URL if underlying generation matches a cached one).

• Cache TTL

  • Default: 24 hours
  • Can be extended via custom_cache_duration_seconds query parameter on generation calls.

9.5 CDN Delivery & URL Expiry

• CDN-Backed URLs

  • All image_url and video_url links point to a global CDN for low-latency delivery.

• URL Expiry

  • Default expiration: 7 days
  • Extendable via the optional url_ttl (in seconds) parameter on generation requests.

9.6 Best Practices for High Throughput

1. Batch vs. Single Requests

   • Aggregate many small videos in /v1/batch/video to reduce overhead.

2. Connection Management

   • Use async + webhooks rather than long polling for large jobs.

3. Retry & Backoff

   • On HTTP 429, respect the Retry-After header.
   • Implement exponential backoff for 5xx errors.

4. Parallelization

   • Limit concurrent requests to your rate limit.
   • Use multiple API keys (scoped per project) if needed.

You can control the underlying generator by passing the model parameter in your /generate/frame or /generate/video requests. Available modes trade off speed, stochasticity, and visual fidelity.

6.1 GAN-Based Mode

• Identifier: model=gan

• Overview: Uses a StyleGAN-like backbone extended for video. Leverages Adaptive Instance Normalization (AdaIN) to inject the identity vector into each layer and adds per-frame noise to introduce fine stochastic detail.

• Key Components:

  • Generator: 2D convolutional "synthesis" network with style modulation.

  • Noise Inputs: Per–layer Gaussian noise tensors (n\u209c \u223c N(0,1)) to diversify textures.

  • AdaIN:

    AdaIN(x, f_identity) = \u03c3(f_identity) * ((x - \u03bc(x)) / \u03c3(x)) + \u03bc(f_identity)
    

  • Discriminators:

    • Frame Discriminator: 2D CNN to judge individual frames.
    • Temporal Discriminator: 3D CNN to judge frame sequences.

• API Usage Example:

  {
    "model": "gan",
    "reference_image_id": "ref_12345abcd",
    "control_type": "pose",
    "z_t": [...],
    "resolution": "1280x720"
  }
  

• Pros & Cons:

  Pros	Cons
  Fast inference (1–2 sec/frame at 720p)	May exhibit flicker without strong temporal loss tuning
  Rich, detailed textures via noise inputs	Can require heavier GPU for high-res

6.2 Diffusion-Based Mode

• Identifier: model=diffusion

• Overview: Iteratively denoises a Gaussian noise map into the target frame over S steps. Conditioned on f_identity and z\u209c via cross-attention in a 2D/3D U-Net.

• Key Components:

  • U-Net Backbone:

    • Down/up‐sampling streams with skip connections.
    • 2D convs for spatial, 3D convs or temporal-blocks for short-range temporal modeling.

  • Cross-Attention:

    • Queries from noisy frame x\u209c\u02e2; Keys/Values from [f_identity, z\u209c].

  • Denoising Loss:

    L_diff = E_{\u03b5, t, s} \u2016 \u03b5 - \u03b5_\u03b8(x\u209c\u02e2, f_identity, z\u209c) \u2016\u00b2
    

• API Usage Example:

  {
    "model": "diffusion",
    "reference_image_id": "ref_12345abcd",
    "control_type": "text",
    "z_sequence": [...],
    "format": "mp4"
  }
  

• Pros & Cons:

  Pros	Cons
  Highly stable subject consistency	Slower (10–50 sec/frame at 720p)
  Better handling of complex scenes & lighting	May require tuning of step count S

6.3 Transformer-Based Mode

• Identifier: model=transformer

• Overview: Treats video as a sequence of spatio-temporal tokens. A Vision Transformer encoder–decoder uses self-attention to model long-range dependencies, with cross-attention to integrate f_identity.

• Key Components:

  • Patch Extraction: Splits each frame into P×P pixel patches, linearly projected to token vectors.

  • Spatio-Temporal Blocks:

    • Self-attention across both time (frames) and space (patches).
    • Cross-attention layers attending to [f_identity, z\u209c].

  • Output Head: Reconstructs patches into image grid.

• API Usage Example:

  {
    "model": "transformer",
    "reference_image_id": "ref_12345abcd",
    "control_type": "latent",
    "z_sequence": [...],
    "fps": 24
  }
  

• Pros & Cons:

  Pros	Cons
  Excellent temporal coherence over long clips	Highest memory usage (TPU/GPU required)
  Unified handling of spatial & temporal context	Slower than GAN but faster than diffusion

To guarantee that generated frames faithfully preserve the subject's appearance and transition smoothly, the system employs multiple specialized losses. Clients do not directly interact with these losses but can tune high-level consistency parameters via optional query fields in generation calls (e.g. identity_strength, temporal_strength).

7.1 Identity Preservation

Ensures each frame's subject matches the reference image in appearance.

• Feature-Space Loss Compare deep features of masked regions between I_ref and generated frame I_t using a pretrained VGG network:

  $$ L_{\text{identity}} = \sum_{l} \big\lVert \phi_{l}(I_{\text{ref}}\odot M_{\text{ref}}) - \phi_{l}(I_{t}\odot M_{t}) \big\rVert_{2}^{2} $$

  • $\phi_{l}$: activations at layer $l$.
  • $M_{t}$: predicted mask for frame $t$.

• Face Recognition Loss (for facial subjects) Use a lightweight face-recognition embedding $e(\cdot)$ and enforce cosine similarity:

  $$ L_{\text{face}} = 1 - \cos\big(e(I_{\text{ref}}),,e(I_{t})\big) $$

7.2 Temporal Coherence

Encourages smooth transitions and discourages flicker or jumpiness.

• Optical-Flow Warping Loss Compute forward flow $F_{t\to t+1}$ via a flow network, warp $I_{t}$ and compare to $I_{t+1}$:

  $$ L_{\text{flow}} = \big\lVert \text{Warp}(I_{t},,F_{t\to t+1}) - I_{t+1}\big\rVert_{1} $$

• Temporal Discriminator Loss A 3D-CNN discriminator $D_{\text{temp}}$ distinguishes real vs. generated clips:

  $$ L_{\text{temp}} = \mathbb{E}\big[\log D_{\text{temp}}({I_{t}}{\text{real}})\big] + \mathbb{E}\big[\log(1 - D{\text{temp}}({I_{t}}_{\text{gen}}))\big] $$

• Sliding-Window Attention Transformer-style self-attention over neighboring frames:

  $$ \hat h_{t} = \mathrm{Attention}\big(Q_{t},,K_{t-k:t+k},,V_{t-k:t+k}\big) $$

  – implicitly enforces consistency over a window of size $2k+1$.

7.3 Domain-Specific or Perceptual Losses

Additional optional losses depending on subject type or client needs.

Clients can adjust these via optional fields in generation requests:

{
  "reference_image_id": "...",
  "model": "diffusion",
  "control_type": "text",
  "z_sequence": [...],
  "identity_strength": 0.7,
  "temporal_strength": 0.8,
  "adv_weight": 0.5
}

Below are quickstart snippets in cURL, Python, and JavaScript. For full reference, see our GitHub repos and PyPI/npm packages.

10.1 cURL Examples

10.1.1 Upload Reference Image

curl -X POST "https://api.aicharacter.studio/v1/reference-image" \\
  -H "Authorization: Bearer $API_KEY" \\
  -F "image=@/path/to/subject.jpg"

10.1.2 Generate Pose Control Vector

curl -X POST "https://api.aicharacter.studio/v1/control/pose" \\
  -H "Authorization: Bearer $API_KEY" \\
  -H "Content-Type: application/json" \\
  -d '{ 
        "reference_image_id": "ref_12345abcd",
        "keypoints": [
          { "x": 0.5, "y": 0.1 },
          { "x": 0.5, "y": 0.25 }
        ]
      }'

10.1.3 Render a Single Frame

curl -X POST "https://api.aicharacter.studio/v1/generate/frame" \\
  -H "Authorization: Bearer $API_KEY" \\
  -H "Content-Type: application/json" \\
  -d '{ 
        "reference_image_id": "ref_12345abcd",
        "control_type": "pose",
        "z_t": [0.12, -0.33, 0.77],
        "resolution": "1280x720"
      }'

10.1.4 Start an Async Video Job

curl -X POST "https://api.aicharacter.studio/v1/generate/video" \\
  -H "Authorization: Bearer $API_KEY" \\
  -H "Content-Type: application/json" \\
  -d '{ 
        "reference_image_id": "ref_12345abcd",
        "control_type": "text",
        "z_sequence": [[…], […]],
        "async": true,
        "callback_url": "https://your.app/hooks/complete"
      }'

10.2 Python SDK

from aicharacter import Client

client = Client(api_key="sk_live_abc123")

# 1. Upload image
ref = client.upload_reference_image("/path/to/subject.jpg")

# 2. Generate text control
z_seq = client.control.text(
    reference_image_id=ref.id,
    prompt="A cat playing piano",
    steps=8
)

# 3. Generate video synchronously
video = client.generate.video(
    reference_image_id=ref.id,
    control_type="text",
    z_sequence=z_seq,
    format="mp4",
    resolution="720x720",
    fps=24
)

print("Download at:", video.video_url)

10.3 JavaScript (Node/Vue) SDK

import { AiCharacterClient } from "@aicharacter/studio";

const client = new AiCharacterClient({ apiKey: "sk_live_abc123" });

async function run() {
  // 1. Upload
  const { reference_image_id } = await client.referenceImage.upload(
    "/path/to/subject.jpg"
  );

  // 2. Pose control
  const { z_t } = await client.control.pose({
    reference_image_id,
    keypoints: [{ x: 0.5, y: 0.1 }, { x: 0.5, y: 0.25 }]
  });

  // 3. Render frame
  const frame = await client.generate.frame({
    reference_image_id,
    control_type: "pose",
    z_t,
    resolution: "1280x720"
  });

  console.log("Frame URL:", frame.image_url);
}

run();

Q1: My generated frames flicker—how can I improve temporal stability?

• Solution: Increase the temporal_strength parameter in your generation call (range 0.0–1.0, default 0.8).
• Tip: Use model=diffusion or model=transformer, which natively enforce stronger temporal consistency than gan.

Q2: I keep getting HTTP 429—what's the best retry strategy?

• Honor the Retry-After header in the 429 response.
• Implement exponential backoff (e.g., initial 1 s, multiply by 2 up to a max of 32 s).
• For large jobs, switch to async=true + webhook to avoid holding client connections open.

Q3: My mask endpoint returns 404 even though the upload succeeded.

• Confirm that the reference image's status (from GET /v1/reference-image/{id}) is "processed".
• If status is "pending", wait a few seconds and retry.
• If it remains pending > 60 s, contact support.

Q4: Which control type is best for highly dynamic motions?

• Pose: precise but requires keypoint data per frame.
• Text: easy to specify broad actions ("running," "jumping") and auto‐generates sequences.
• Latent: use for abstract or style-driven motion; less semantically grounded.

Q5: How do I mask out the background in generated frames?

• Generated frames come with optional masks if you append ?include_mask=true to your frame/video URL.
• The mask URL will be in your JSON response under mask_url.

Q6: I need higher resolution than 1280×720—what are my options?

• Supported resolutions: up to 1920x1080 on shared GPUs, and up to 4K (3840×2160) on dedicated A100 instances.
• To request >1080p, include header X-GPU-Type: a100.

Q7: My asynchronous job failed with "ResourceExhausted."

• This indicates GPU memory limits were exceeded (e.g., too many high-res frames).
• Reduce resolution or switch to async=true with a higher-tier GPU via X-GPU-Type.
• If persistent, split into smaller batches.

Q8: Can I reuse existing z_sequence across multiple videos?

• Yes. For reproducibility, store both reference_image_id and the exact z_sequence.
• Ensure you also specify the same seed when using latent control for deterministic output.

Q9: What image/video formats are supported?

• Frames: PNG or JPEG (via output_format).
• Videos: MP4 (H.264) or animated GIF.

Q10: Who do I contact for support or to request new features?

• Email: support@aicharacter.studio

Q11: How do I get access to use the API?

• Send an email to dev@aicharacter.studio to request credentials and onboarding details.
• Or DM us on X (formerly Twitter) at https://x.com/aidotstudio.