Prepare Input Footage and Track

To prepare the footage for the main workflow, use the MASK ACTOR + CAMERA TRACK workflow first.

We created this easy workflow to quickly process clips and generate mask and tracking outputs.

1. At the beginning, you can upload your clip.

2. You can also set a shot number or name that will be used for all outputs.

3. We added an option for activating interpolation. Long clips that don't fit into memory or require extensive processing time benefit greatly from reducing the number of frames that need to be processed by half. The other half will be done generated by interpolation.

4. For easy toggling of the optional workflow sections.

GENERATE MASK OUTPUTS

This section should work automatically in most cases. If you need a different resolution, you can adjust it using the Resize Image node.

GET FIRST FRAME

This section allows you to export the first frame, which is useful when you want to use image-to-image (I2I) workflows.

CAMERA TRACK

This section generates a camera track output for the next workflow. It initially masks out the area where the actor appears to prevent tracking points from being created there. You can adjust the mask size based on your specific needs.

Additionally, we've maintained the option to input vector data, as we had in the creature path workflow.

MODEL LOADERS

This workflow has two model loader groups:

• FP8 Models → Using Kijai's fp8 Wan2.1 14B Models - extremely powerful but very VRAM intensive!

• GGUF Models → Memory-efficient with options for various VRAM capacities.

ℹ️ Note: The active sampler in the SAMPLING section automatically determines which model is used. You don't need to manually disable or enable model groups to conserve memory..

Step 2: Video Input & Resolution

In this section you can set up your input video.

1. Resolution: 720p is recommended for the best quality.

2. Batch Size: Set the number of frames processed per batch.

3. Skip: Offset the start frame of your footage.

4. Frame Load Cap: 81 frames is recommended, as WAN 2.1 was trained on 81-frame data. However, we've had successful results with 40 frames more or less. If you encounter issues, they may be related to frame count.

5. Interpolation: This uses every second frame of the input video and interpolates the skipped frames afterward to reduce the total number of frames needed to be generated for the full range.

6. Upload Video Inputs

   1. Masked Actor: Upload your actor against a 50% gray background, which yields the best results with VACE. Using other backgrounds can cause bugs like brightness changes.

   2. Actor Alpha Mask: Needed for the VACE Encoder Nodes.

   3. External Camera Tracking:

      • For Unique Perspective we used After Effects. We exported the tracks by creating red boxes in 3D space as reference points. VACE can interpret these boxes for camera tracking.

      • Alternatively, use our MASK + TRACK workflow — which includes a CAMERA TRACKING section.

Step 4: Background / Reference Image

You can upload an image to use as a reference background for the video. If you prefer to generate the video solely from text prompts, this step can be disabled by toggling the switch to 2.

Step 5: Start Frame Options

There are three options for the start frame:

1. Actor + Background Image: Places the masked actor on top of the background image for maximum adherence to the reference image.

2. Background Image Only: Uses only the background image, which is useful when the actor shouldn't appear in the initial frame.

3. NO Startframe: Gives WAN more flexibility to generate the scene environment.

Step 6: Control Images

This section creates the Control-Images for VACE, primarily to provide data about camera movement and person positioning.

You have 4 options:

1. Track Only: Passes the camera track loaded in the VIDEO INPUT section.

2. Track + Pose: Improves interaction with the environment and enhances overall tracking accuracy.

3. Canny + Pose: Ideal for pre-designed backgrounds. For example, scenes created in Blender and exported as a video.

   

4. Input CN Video: Save and re-upload output, recommended for memory-intensive workflows to save time and resources.

Step 7: Sampling and Fine-Tuning

Select your preferred sampling method. When the FP8 MODEL SAMPLER is active, the workflow will use Kijai's fp8 models. When the GGUF MODEL SAMPLER is active, it will use GGUF models. Ensure that only one sampling method—either FP8 or GGUF—is active at any given time.

1. Fast Groups Muter: Use this to toggle consecutive iterations on or off. This feature allows you to fine-tune each iteration's seed values and prompts before moving on to the next batch.

2. VACE Strength: Adjust this value to fine-tune your results. The second VACE Encode processes the actor footage, determining how closely the appearance matches the input versus how well it integrates into the background. If you notice jitter at the maximum value of 1, reducing it even slightly can help eliminate this issue.

3. Dilate/Erode Mask: Increase this value to reduce the actor's mask when artificial outlines appear.

4. Prompts: Describe clearly what is happening in the shot.

5. WanVideo Sampler Steps (FP8 only): With the lightx lora, 6 steps is optimal. We recommend sticking with 6 steps in most cases, as higher values may reduce the quality of the inserted person or create artifacts.

   • When working with fp8 models, you can create quick low-quality preview generations using just 2 steps—in our testing, these previews closely matched the final results, allowing you to efficiently test different seeds. Note that this tip doesn't apply to the GGUF sampler.

6. As always, experiment with different seeds to find the best possible animation.

7. This Video Combine node allows you to check the results of the current batch.

8. Here, all the generated videos up to this iteration are merged into a single video clip.

Additional Tips

Video sampling previews are now available, allowing you to see the result in real-time!

Set the Preview method in the ComfyUI Manager to Auto.

In the settings go to the 🎥VHS tab and enable Display animated preview when sampling. Be sure to have the latest version of the node pack.