Image and video annotation services mainly involve labeling and tagging of objects, actions, attributes, elements or other relevant information within the media. This process is done to train artificially intelligent or machine learning models on supervised training data that in turn replicate annotations without human supervision through a process called model-assisted labeling. The whole process helps AI models to perform various tasks similar to a human eye perspective such as recognizing patterns, faces, objects and making predictions. To perform this task, either annotators who are trained in related tools and techniques can be hired, or assistance can be taken from, Open-source platforms like CVAT, labelImg, crowdsourcing platforms and service providers that develop automated annotating solutions. It is necessary to follow quality control measures while drawing bounding box, assigning object the pre-defined nomenclature etc. to meet the requirements of the algorithm concerned, and thus, it may require expertise around type of annotation, modality of data and domain-specific knowledge for format in which annotated data will be stored.
The total time required for annotation depends on the number of objects and key points to be annotated, amount of data needed to train deep learning model, number of classes to assign the objects to, and the complexity of the annotation process, per say, a minute of video with 30 frames per second and 1800 sequentially arranged images. Outsourcing annotation to service providers can allow organizations to allocate their resources to other creative tasks, save their time and helps them gain access to expertise, especially if the project duration is short. The global market value of image and video annotation services is likely to increase at a CAGR of 17% during the forecast period 2020-2030. Let us go through the intricacies of these services, as we explore their methods, benefits, use cases and challenges.
The above diagram shows video labeling service for traffic analysis
Methods of Image & Video Annotation Services
There are many types of annotations and respective methods used for implementing the same which are further discussed below:
Bounding Box
This type of straightforward image and video annotation involves rectangular regions drawn around an object, person or vehicles within the image or frames of the video concerned, such that the algorithm can localize and classify them to pre-defined levels. In a particular use case of training models with such datasets, that is image classification, simple text labels, class numbers/IDs or one-hot code encodings can be used. The common representations of bounding box coordinates in the image are {x1, y1, x2, y2}: upper left corner – (x1, y1); lower right corner – (x2, y2) and {x1, y1, width, height}. These annotations are not ideal for objects with irregular orientations and rotating ones, in addition, these include background pixels along with the objects when they are closely placed in the image.
Polygon
In this type of image labeling services, irregular shapes are drawn around versatile shapes and sizes of objects that usually has multiple vertices, thus providing more precise boundaries and making datasets suitable to train models that perform instance, image segmentation for intelligent video analytics and scene text recognition. For example, class names are assigned to objects detected and recognized in an image, where class ID and coordinates of the polygon are set as the ground truth. Polygonal masks can be vectorized easily and occupy low space for better accuracy. The common representations of such annotations have a specific sequence of interior, exterior x and y coordinates that form the polygon in the image, where overlapping objects may pose a challenge.
Segmentation
Semantic segmentation is a method that assigns a class label to every pixel in a particular image or video frame, that is, the image is segmented into various regions and corresponding object categories and annotated as binary masks or segment masks marked by class ID. It is useful in generating datasets used for models deployed in autonomous driving and medical imaging for detecting circularity, area, size and localization of cells, tissues etc. The output is a typical pixel-wise mask in .png format where each color corresponds to a class or .json format with bitmap objects encoded as base64 strings.
Instance segmentation, on the other hand, distinguishes individual object instances that are of the same class in an image by assigning unique identifiers. This means algorithms can differentiate between objects of same class label when trained on such type of annotated datasets by image labeling services. Another type is panoptic segmentation which forms the conjunction of both types, as the algorithm needs to segment object categories while detailing out instance level segments. In this case, every category and object instance gets assigned a specific segment map.
The above diagram shows different types of image labeling services
Keypoint
In this type of video and image annotation, specific landmarks and points on the object are labeled. Such datasets are used for human pose estimation in fitness apps, object tracking, high pattern recognition, aerial view and monitoring of forests, weapon reserves, parking lots, mood analysis, facial recognition, deepfake realistic morphing, face replacement using landmark features like eyes, eyebrows, lips, nose, face boundary etc. It requires extensive and diverse set of images captured from various angles under considerable illumination without shadows, hidden object parts, and cluttered backgrounds for accurate annotation and model execution.
Text
Text annotation involves labeling captions, signs, labels etc. present within the image. These annotated datasets are applied in optical character recognition, scene understanding and intelligent document processing.
Line and Curve
Linear and curved structures within the image or video can be annotated by image annotation company, for example, roads, trajectories, water body course, boundaries, etc., and it is therefore, used in algorithms that detect roads, marked lanes, object motion, as well as in warehouse robotics and autonomous vehicles. Line annotation delineates boundaries by marking lines and splines to pinpoint small, narrow features within the image. Certain challenges associated with line annotation are annotation subjectivity or ambiguity as multiple annotators draw lines, annotating complex shapes, and varying image quality with low resolution or high level of noise. Another type is polyline annotations that are basically a set of lines drawn across the input image and are quintessential in use cases like lane detection.
Emotion
Emotion annotation is crucial for applications like sentiment analysis, content moderation and affective computing, in which expressions and sentiments by customers, interviewees etc. can be conveyed by the algorithm trained on such annotated datasets.
3D Cuboid
When object detection masks are used in three-dimensional planes or performed on 3D data with depth, distance and volume, the process involved is called cuboidal annotation. They are used in harvesting robotic arms and motion, labeling anatomical structures in medical scans, LiDAR, urban planning, retail AR/VR, manufacturing quality control, terrain analysis and disaster response within SAR images etc.
Use Cases of Video & Image Annotation Services
Healthcare
Image and video annotation services can be used to annotate CT scans and MRI image datasets. These annotated images can be used for training AI/ML models that run COVID-19 diagnostics. The annotated datasets can also be used for face mask detection in restricted areas to enforce safety measures. EEG and ECG scans and tumor images can be labeled, and datasets can be fed to AI models to depict growth patterns, anomaly identification, organ segmentation, thus aiding radiologists and surgeons in detection and making decisions related to treatment effectiveness. For example, remote surgical procedures with Proximie.
Agriculture
In the agricultural industry, image and video annotation services can be used to label pest infested plant aerial images, satellite imagery, and train models to predict their presence and severity as well as detect and localize such plants. The annotated datasets provide insights to farmers and government into yield, statistics related to crop growth, ripeness, land use, resource monitoring, deforestation, environmental changes etc. for making informed decisions on precision farming, sustainable agriculture, and conservation efforts. This occurs without manual surveillance in the case of automated harvesting and helps to assess land cover classification and urban infrastructure development planning. Models trained on labeled datasets can characterize different livestock animals and help dairy farmers in tracking the well-being of these animals by predicting their health status.
Automotive
Images annotated by image annotation services can help autonomous vehicles to identify safe navigation elements like vehicles, pedestrians, animals, traffic signals, obstacles, road signs, cyclists, driving lane markings, driver behavior, and much more. Models trained on traffic image and video line annotation datasets, can gain input from security cameras placed in the area or city and provide quick analysis of traffic flow, road conditions, congestion, blocks etc. for efficient roadways management and safer autonomous driving by syncing with collision braking system. A few examples include Tesla’s autopilot system, Volvo’s automatic braking system to avoid collision, Germany’s smart traffic management system. Annotated datasets allow models trained on them to predict and detect parking occupancy and available slots for enhanced parking guidance.
Retail
Annotated images formed by image labeling services can train inventory tracking models for various retail technology solutions such as consistent inventory management and product placement analysis. Such labeled data can help models to recognize features of the products, detect customer reactions, footfall, queues, shoplifting activities, perform visual searches with respect to attribute annotation and recommend products to users in retail stores and e-commerce platforms. In manufacturing and warehousing, line annotation can be used to train models that help identify paths for material handling robots, detect defects, irregularities in quality control and maintain efficient operations.
Security
Labeled datasets (facial annotations etc.) are used to train models for video analytics surveillance to detect people, fingerprints, loitering vehicles, unreported road accidents, natural disaster detection, suspicious objects like sharp metals, firearms etc. in public places like airports, metros, stadiums, vandalism, abnormal behavior, in restricted areas such as near defense quarters, drug pharmacies, ATMs, ship docks, borders, through virtual fencing and detection capabilities.
Social Media
Video and image annotation services can be used for content moderation on social media and other online platforms. Annotating instances within images and video with inappropriate comments or offensive or harmful content can be flagged by models trained on such datasets. This helps in enforcing community, forum or platform guidelines and assuring user safety. In the entertainment, media, tourism and hospitality industries, precise real-world scene annotations are used to facilitate augmented reality and virtual reality experiences in various games and social media handles by such organizations. These services can also assist in developing models that can translate sign language into text and speech.
The above diagram shows video labeling service for AR-based application
Benefits & Challenges of Video & Image Annotation Services
Benefits
- Better Performance: As per the concept of Garbage in, garbage out, a properly labeled high-quality input training data can enhance the quality of output or predictions made by the machine learning model as they will be able to recognize objects during object-based image analysis, attributes, actions etc. relevant information within the images and videos.
- Data Generation: These services are necessary to generate large volumes of training data that have been annotated for supervised AI/ML algorithms.
- Multiple Applications: The labeled datasets generated through these services can be used in various industries such as for annotating medical images in healthcare, product recognition and buying pattern labeling as retail store solutions, object tracking in security field, crop monitoring in AgriTech etc.
- High Efficiency: By outsourcing data annotation tasks, organizations can redirect their technical resources to other creative tasks like algorithm development. Dedicated teams streamline the entire data labeling process efficiently with sophisticated tools and expertise, so that organizations can save time and costs for tasks such as OpenCV face recognition.
Challenges
- Errors: If the data annotation process is not performed accurately by experts that follow quality control standards along with multiple cross-checks, reviews, flagging or service provider organizations with ample experience of sharing and storing annotated images, there is likely a possibility of mislabeling and the same is replicated by the model. Detailed context on challenges faced during specific annotations can be read under the methods of annotation section.
- Expensive: Finding the mislabeling instance in the entire dataset, frame of video or finding an object out of thousands of objects in an image to relabel and train the model again by acquiring labeling equipment can cost a lot, per say a part of the entire process needs to be re-enacted. Image and video annotation services can assist organizations to scale cost-effectively while ensuring quality through flexible pricing models, sorting relevant images and optimal resource allocation as a part of cost management measures, for example, costs involved in maintaining a smart parking management system.
- Subjectivity: The entire process can take days to months as per volume, occlusion, low-quality images, detail level required, data complexity or ambiguity. If not interpreted and labeled by domain-specific expert annotators with the requisite tools, well-defined workflows, and edge case handling experience, it can delay project timeline due to discrepancies and prove to be quite resource intensive.
- Data Privacy: Labeling sensitive or personal data such as medical images, buying history pattern from financial records etc. may raise concerns around privacy. This risk is mitigated by annotation service providers who follow secure annotation practices and adhere to data protection regulations.
Key Takeaways
We discussed various methods of video and image annotation services such as bounding boxes, semantic segmentation, classification etc. that render various use cases in industries such as robotics, autonomous vehicles, aerial imagery, robotics and more. KritiKal Solutions is a leading video annotation company that collaborates with organizations and provides image and video annotation services and quality control, sample datasets, versioning, model integration, customization and adaption, data augmentation and transfer learning. Please mail us at sales@kritikalsolutions.com to avail our services. It is likely that advances in deep learning would continue to drive innovation in annotation for more accuracy, efficiency and versatile solutions.
Keshav Goyal holds the position of Senior Software Engineer at KritiKal Solutions. He is a self-driven Full stack developer with experience in Java, Nodejs, Python, Django Rest framework, AWS, ReactJS, SQL, MongoDB, Elastic Search etc. With his quick-learning and team player abilities, he has helped KritiKal deliver various projects to some major clients.