PRISM

Predictive Recomposition vIa Semantic Latent DecoMposition
for View-invariant Video Representation Learning

Youngchae Chee*, Hosu Lee*, Sungjune Park, Junho Kim†, Yong Man Ro†
_{*equal contribution, †corresponding}

---

PRISM is a video encoder that captures viewpoint-invariant action semantics, matching the same action across egocentric and exocentric views. It decomposes each video into view-invariant and view-variant latents and recomposes them across videos under language supervision, which stays semantically valid beyond the co-occurrences observed in training.

_{Watch the full 2-minute demo on the project page}

Results

Cross-view semantic alignment

Method	EgoExo4D Retr. ego→exo	exo→ego	avg	Recog. top-1	Skill	EgoExoLearn Assoc. avg	Antic. avg	Skill
Image-Language Model
CLIP	19.11	12.24	15.68	10.49	54.93	15.82	38.70	73.48
SigLIP2	35.08	19.72	27.40	13.86	55.57	26.6	64.60	76.03
Video-Language Model
TimeSformer	6.68	6.95	6.82	5.18	51.58	16.32	64.19	75.47
InternVideo	23.64	20.67	22.16	13.67	52.58	26.20	65.16	68.88
Ego-Centric Methods
EgoVLP	29.33	13.47	21.40	20.33	54.37	30.20	60.65	69.04
LaViLa	34.91	12.02	23.47	26.43	54.10	27.20	62.83	68.44
Cross-View Methods
ActorObserverNet	29.50	24.85	27.18	15.70	54.10	11.64	56.13	68.59
VI Encoder	29.53	24.40	26.97	14.85	53.83	-	-	-
EgoInstructor	46.04	31.68	38.86	24.15	54.73	-	-	-
SUM-L	47.14	32.77	39.96	24.83	55.10	4.64	45.50	65.31
ViewpointRosetta	58.14	47.21	52.68	34.47	55.82	32.32	62.14	73.70
PRISM	75.89	50.27	63.08	41.93	55.28	43.86	69.46	68.53

Fine-grained temporal modeling on AE2

Method	AE2 videos	Frame retr. mAP@10	Phase order Kendall's τ	Phase class. F1	Phase prog. R²
Random		52.28	0.004	33.27	−0.069
Trained w/ AE2 videos (in-domain)
ActorObserverNet	✔	44.82	0.002	34.51	−0.052
TCN	✔	49.37	0.046	44.71	−0.227
CARL	✔	51.81	0.025	45.42	−0.124
TCC	✔	66.17	0.400	58.45	0.322
GTA	✔	68.08	0.464	67.77	0.322
AE2	✔	73.20	0.562	74.47	0.480
Trained w/o AE2 videos (zero-shot transfer)
ResNet-50	✘	48.69	0.025	43.41	−1.215
CLIP	✘	47.44	0.047	44.23	−1.212
SigLIP2	✘	45.56	0.020	43.91	−1.322
SUM-L	✘	45.80	0.110	50.93	0.030
ViewpointRosetta	✘	54.17	0.047	46.93	−0.150
PRISM	✘	70.53	0.601	73.57	0.647

Remaining benchmarks and ablations are in the paper. Check it out!!

Install

The project uses uv; the environment is created and synced automatically on the first uv run.

uv sync                  # core (training + inference)
uv sync --extra serve    # + vLLM, only to host the recomposition LLM

The frozen backbones (google/siglip2-so400m-patch14-384, Qwen/Qwen3-Embedding-0.6B) download from the Hugging Face Hub on first use.

Quickstart

The released checkpoint lives at litcoderr/prism and loads straight from the Hub.

import torch
from transformers import AutoImageProcessor
from prism import PRISMModel

model = PRISMModel.from_pretrained("litcoderr/prism", dtype=torch.bfloat16).eval().cuda()
proc = AutoImageProcessor.from_pretrained(model.config.vision_backbone_name)

frames = [...]                                              # list[PIL.Image], sampled at 4 fps
pixel_values = proc(images=frames, return_tensors="pt").pixel_values[None]
pixel_values = pixel_values.to("cuda", torch.bfloat16)      # (1, T, 3, 384, 384)
valid_mask = torch.ones(pixel_values.shape[:2], dtype=torch.bool, device="cuda")

emb = model.encode(pixel_values, valid_mask)                # (1, 512), L2-normalized

The clip embedding is the mean-pooled z_vi stream, so the cosine similarity between two embeddings is a cross-view semantic similarity. model.encode_streams(...) returns the per-frame z_vi and z_vv instead.

The Hub repo carries its own copy of the modeling code, so you can load the model without cloning this one:

from transformers import AutoModel
model = AutoModel.from_pretrained("litcoderr/prism", trust_remote_code=True).eval().cuda()

Encode a manifest of clips

CHECKPOINT_DIR=litcoderr/prism \
MANIFEST_JSON=clips.json \
VIDEO_ROOT=/path/to/videos \
OUTPUT_PATH=features.npz \
bash scripts/encode.sh

clips.json is a list of clips; features.npz holds features (N, 512), item_ids (N,), success_mask (N,):

[{"item_id": "c0", "video_uid": "vid123", "start_sec": 0.0, "end_sec": 8.0}]

Training

Reproduces the released checkpoint: SigLIP2-so400m / Qwen3-Embedding-0.6B, 6 epochs, batch 4 × 7 GPUs, constant-warmup lr 7e-5, EMA 0.998.

1. Inputs. Per-clip captions are a provided input: a JSON mapping each segment's annotation_id to its view-invariant (T_vi) and view-variant (T_vv) descriptions. action_caption / context_caption are accepted as aliases.

{
  "<annotation_id>": {
    "view_invariant_caption": "a person slices an onion with a chef's knife",
    "view_variant_caption":   "first-person view in a home kitchen, overhead light"
  }
}

The split JSON is a list of segments with annotation_id, video_uid (or video_path), start_sec, end_sec. The released checkpoint uses captions generated by Gemini 3; an audit with two LLM judges and five human annotators measured 95.2% view-invariant words in T_vi and 99.1% view-variant words in T_vv. Swapping the captioner for Qwen3-VL moves cross-view and temporal alignment by −0.6 and −0.3.

2. Serve the recomposition LLM (on its own GPU):

COMPOSE_GPUS=7 bash scripts/serve_composer.sh        # vLLM @ 127.0.0.1:8765

3. Train:

SPLIT_JSON=splits/train.json \
CAPTIONS_JSON=captions.json \
VIDEO_ROOT=/path/to/videos \
OUTPUT_DIR=./prism_run \
NUM_GPUS=7 TRAIN_GPUS=0,1,2,3,4,5,6 \
COMPOSE_API_KEY=EMPTY \
bash scripts/train.sh

All paths, the composer endpoint, and the API key are environment variables, never hard-coded. The final weights-only checkpoint is written to OUTPUT_DIR/final.

Citation

• TODO: Add citation once published.

License

MIT. The frozen backbones keep their own licenses (SigLIP2, Qwen3-Embedding); the evaluation datasets keep theirs.