Video has become the new standard for Computer Vision applications. An image shows what is present in the scene; a video shows what is changing and what remains constant over time. Both are important for applications such as autonomous vehicles, surveillance security, and sports analytics, where timing and motion can significantly impact success or failure.

Demand continues to rise: Grand View Research projects that the video annotation tools market will reach USD 1.9 billion by 2030, growing at a 26.6% CAGR from 2024 to 2030.

Video annotation in machine learning converts raw footage into structured ground truth so models can learn from sequences -- not individual images. When compared to image labeling, video dataset labeling includes an additional task: maintaining consistency of identities of objects across frames even when objects are blurred, overlapping, and/or reappearing.

High-quality labels will produce high-quality models. If your boxes are unstable or your masks fail to identify all edges of the object, mAP, precision and recall will generally be impacted due to the model's reliance upon variable targets during training.

Here are the six label types usually found in video annotation projects. They address the key questions that teams working with video annotation in machine learning face routinely.

Best methods of video annotation for AI projects

AI projects harness a range of video annotation techniques. What is used for any given project depends on approach, goals and the user's experience.

2D Bounding Boxes: The Foundation of Object Detection

Two-dimensional boxes surround an object in each frame and define it via x and y coordinates. As they are very efficient and can provide good detection and tracking results, 2-D boxes are typically the first step taken in video annotation. Bounding boxes are touted to be a low-cost solution to detecting and tracking objects across time in automotive, retail and surveillance.

Video introduces issues of continuity. Most teams do not label every frame. Instead, they label key frames and allow the software to fill-in the missing frames. CVAT utilizes this workflow through their Track Mode: you manually edit a few key frames where motion changes, and the software updates the intermediate frames accordingly.

Polygonal Annotation: Capturing Complex Geometries

Polygons work faster than rectangles. But they use multiple points to trace an object's outline and so the work needs more precision. This produces fewer background pixels, thus reducing training noise in highly populated scenes. Polygonal annotation provides a higher level of precision for irregular shapes and is related to drone mapping, self-driving vehicle navigation and agricultural analytics.

Polygonal annotation is the preferred method when high levels of accuracy are required to identify boundaries, such as pedestrians in various poses, vegetation edges, irregular shaped machinery, etc. Additionally, polygons can help when objects overlap frequently and collision occurs among bounding boxes.

Polyline Annotation: Defining Paths and Boundaries

Polylines are basically linear structures made of connected points. They help identify lanes, curbs, sidewalks and similar path characteristics. CVAT lists polyline as a Core Shape Type and notes that polyline annotation works well for Road Markup Annotation.

Video presents challenges for polylines. Lane lines for instance may curve, move, disappear and reappear. You need to make temporal adjustments and specific rules for video to handle such gaps and occlusions. When labels flow into a training pipeline, we can use Labelbox for importing video annotations and polyline creation in their Video Editor.

Semantic & Instance Segmentation: Pixel-Perfect Context

Segmentation is a way to assign labels to individual pixels in an image. Semantic segmentation labels each pixel according to class (road, building, etc.). Instance segmentation identifies each instance of an object class (two cars side by side). Semantic segmentation means labeling each pixel for context and describes instance segmentation as the ability to distinguish each instance.

When the context of the scene matters just as much as detection, and there will be overlap between the various classes of objects, teams will select segmentation. Effort is the tradeoff for selecting segmentation; therefore many projects label a subset of frames, train a helper model to assist in the task, and then manually correct the draft masks created by the helper model.

Keypoint & Landmark Annotation: Tracking Pose and Motion

Keypoints pinpoint exact locations on objects, such as body joints or facial features. Landmarks follow the same principle and include additional points that map out an entire outline or skeletal framework. These annotation methods power motion and expression analysis. Common applications include gesture control interfaces, pose estimation for athletic performance, and other systems that need to understand how objects move and change position.

Keypoints are useful when movement is the desired signal. They allow models to understand intent cues such as reaching, turning, and falling. As with other types of annotations, video creates an issue of identity management, and teams develop rules for when points should be assumed missing during occlusions and how to continue tracking once an object reappears.

3D Cuboids: Depth Perception in a 2D Frame

Three-dimensional cuboids provide both volume and orientation to a 2-D frame. Three-dimensional cuboids inform a model how an object occupies space, thereby aiding in distance and clearance. Three-dimensional cuboid annotation captures depth and spatial orientation, and examples of 3D cuboid annotation include autonomous vehicles and warehouse robots.

Three-dimensional cuboids appear when systems require detecting objects and planning their own movements around those detected objects. There is great value in testing three-dimensional cuboids for quality assurance (QA), since minor pose errors can be compounded in downstream planning.

Can video annotation be machine-assisted, and which tools help?

The majority of teams make use of machine assistance along with human review.

The standard process is: annotate keyframes, propagate, and correct. Time savings from keyframes become clear when you look at CVAT's track and interpolation features. HitechDigital offers support for a variety of video label types and supports importing formatted data into a production workflow.

The Intersection of Human Expertise and Automation

You'll find that these six annotation types cover all the core techniques teams use when they go about labeling large video datasets. Boxes work well for most detection tasks. Polygons and segmentation give you precise boundaries. Polylines help you mark lanes and paths. Keypoints show you pose and intent. Cuboids add depth and orientation to your annotations.

Therefore, teams desiring to achieve scalability must ensure clear labeling specifications and quality assurance (QA) requirements are met as equally as selecting the appropriate tool, and thus many teams utilize specialized video annotation services to maintain consistency and throughput in real-world deployment environments.

Top 6 Types of Video Annotation Used in AI Projects