Over 10 years we help companies reach their financial and branding goals. Engitech is a values-driven technology agency dedicated.

Gallery

Contacts

411 University St, Seattle, USA

engitech@oceanthemes.net

+1 -800-456-478-23

Twitter Facebook-f Pinterest-p Instagram

Guided Image Capture for Clinical Subjects

Guided Image Capture for Clinical Subjects
Category:
AI / FMCG Beauty Care

The Problem Statement

Users of cosmetic products or clinical subjects regularly take selfie images for skin analysis and tracking the same over time. Most of the time the captured images are not in proper pose or lighting, thereby making the images useless for analysis. There is a need for a guided image capture app to guide users while capturing selfie images from a mobile device with the same head pose, distance, expression, occlusion, and other conditions. Such an app helps users in capturing an optimal selfie image. which can be further used for any clinical analysis. 

The Solution

We developed a web browser-based solution for guiding the user and auto–capturing of the image once all the predefined conditions are met. Once the image is captured, it could be either uploaded to the server for further analysis or the user can retake the selfie again. High–resolution image captured via browser on iPhone, iPad, and Android phone at a reasonable FPS. All the AI models run on the mobile device (edge computing) with a reasonable speed. 

Skin analysis features

3D Face pose estimation and user guidance

This feature helps in detecting the 3D pose of the face and helps the user via 3D animation to keep the face in the proper pose that meets the guidelines.

Audio cues and animation provided for user guidance

We have provided creative animations and audio cues to the user to change the face pose or distance to the camera, etc.

Low light, bright light, and directional light detection

Lighting on the face either low or bright light, causes issues during the skin analysis. So, this feature helps in detecting not only the low or bright light but also the directional lightning falling on the face. User will be indicated to move to a different location with proper illumination to avoid this scenario.

Auto capture and Blurry image detection

Once all the capture parameters are met, face image will be auto captured, which will be checked for a blurry image in case of any movements of mobile during the image capture.

Obstacle detection on face

The obstacle detection feature helps in detecting the presence of obstacles on face, like masks, hair, goggles/specs, etc., which can create problems during the skin analysis. Once the obstacles are detected, users will be informed about the obstacles and remove them for image capture.

Visualization

Once the user hits the app URL in their mobile browser, the app asks for the camera permission, after which the app shows certain tips for efficiently capturing the selfie image. The app shows a live feed of the person in front of the camera with a face mesh and shows 3D face pose estimation, that is yaw, pitch, and roll in terms of animation. The app also gives audio cues and visual guidance by displaying messages like face tilted left/right, face is too close/far, keep face within oval, keep neutral expression, remove glasses/mask, etc. 

Mobile app, AI model optimization & porting

This web browser-based guided selfie app runs on the Android, iPhone and iPad. Different computer vision algorithms and AI modules within the app were optimized to run smoothly on the end user’s mobile browser. The app was hosted on the cloud, and once the user hit the app URL in the browser, it asks for camera permission. After successful access to the camera, the app starts downloading the AI model in cache and starts loading the required AI models in the background. 

The app uses the Google Media Pipe library for 3D Face pose estimation. The source code and AI models were ported from Python to JavaScript to run smoothly on the end user’s mobile browser. We optimized the AI models and computer vision source code to run the app smoothly. App will process the live feed at slightly lower resolution and once all the criteria for capturing the optimal frame are met, we capture the highest resolution image of the subject captured by that phone model at that moment. 

Benefits Delivered

  • Swift pre-processing of captured images and accurate skin analysis.
  • Detailed and easy-to-understand skin score reports for customer retention.