Real-time Facial Detection and Validation

Using computer vision and AI to recognize facial expressions and enhance vehicle safety

The combination of computer vision and AI is a powerful tool in detecting a person’s state of mind through facial expressions. It is especially useful in an automotive scenario where it can assist or alert the driver of a vehicle in an emergency.

Detecting and recognizing facial expressions

Facial expressions are a unique way of exposing feelings or thoughts. As a form of nonverbal communication based completely on the human face, they are one of the easiest, most reliable ways of detecting the emotional condition of an individual.

Analyzing the expressions of a person driving a vehicle represents a unique application for real-time computing. To deal with drivers who fall asleep or lose awareness at the wheel, quick, accurate detection of facial expressions can play an essential role in road safety.

Consider a model for detecting facial expressions in which a camera mounted in the vehicle continuously monitors the facial expression of the driver. The model must recognize and label the mental state associated with the facial expression. For an app based on such a model, the following test scenarios assume that almost all humans use almost the same facial expression for certain common feelings.

Note: Not all drivers will undergo the facial changes for each of the expressions described below. However, the combination of most of them is useful in detecting the expression of the driver accurately.

Test scenario 1: Happiness

Facial changes that indicate the happy state include:

• Corners of the mouth raised upward
• Cheeks raised
• Eyes partially closed
• Wrinkles in skin around the eyes
• Teeth exposed with an open mouth

As shown in the upper left of the green square in the photo below, when the app detects these facial changes, it should label the expression "Happiness".

Facial expression sample: Happiness

Test Scenario 2: Sadness

Sadness is the opposite emotion to happiness. In this state of mind, the person feels low and helpless. There is more chance for drivers to lose control of the vehicle while feeling sad due to absence of mind. Some secondary feelings related to this state are suffering, hurt, despair, pity and hopelessness.

Facial changes that indicate the sad state include:

• Corners of the mouth lowered
• Inner corners of the eyebrows raised
• Eyebrows wrinkled
• Lowered upper eyelid
• Jaw raised
• Slightly pouting lower lip
• Closed eyes

When the app detects these facial changes, it should label the expression "Sadness".

Facial expression sample: Sadness

Test Scenario 3: Anger

Anger is one of the most dangerous emotions because it causes a gradual increase in heart rate and blood pressure. It should be accorded high importance since it is directly related to the health condition of the driver. People should avoid anger while driving a vehicle because they may be susceptible to feeling irritated or enraged. Some secondary feelings are annoyance, frustration and hatred.

Facial changes that indicate anger include:

• Eyebrows pulled down together
• Teeth shut
• Lips held tight with corners down
• Brow furrowed (i.e., vertical lines or wrinkles near eyebrow)
• Eyes opened wide in a hard stare
• Chin raised
• Nostrils flared

When the app detects these facial changes, it should label the expression "Anger".

Facial expression sample: Anger

Test Scenario 4: Fear

Fear is a reaction to perceived danger. In this mental state there is little chance of the driver losing awareness because human nature dictates that the mind will be active as normal. Some secondary feelings are horror, nervousness, panic, worry and dread.

Facial changes that indicate fear include:

• Eyebrows raised together
• Flat line above eyebrows
• Mouth opened slightly
• Lips stretched horizontally toward ears
• Eyes opened slightly wider than normal
• Wrinkles on the forehead

When the app detects these facial changes, it should label the expression "Fear".

Facial expression sample: Fear

Test scenario 5: Surprise/Excitement

Surprise comes when unexpected things — either positive or negative — happen. As with happiness, the driver maintains presence of mind with little chance losing awareness. Nevertheless, overexcitement due to surprise or happiness can also impair control of the vehicle. Some secondary feelings are amazement, astonishment and shock.

Facial changes that indicate surprise include:

• Mouth opened wide
• Teeth parted
• Eyes opened wide
• Skin below eyebrow stretched
• Eyebrows raised
• Horizontal wrinkles on forehead
• Jaw dropped

When the app detects these facial changes, it should label the expression "Surprise".

Facial expression sample: Surprise

Test scenario 6: Disgust

Disgust is a feeling of intense dislike. It can arise from taste, smell, sound or touch. A driver in a state of disgust is liable to lose awareness at the wheel. Some secondary feelings are irritation, dislike and offense.

Facial changes that indicate disgust include:

• Raised cheeks
• Raised upper eyelid
• Wrinkles on nose bridge
• Wrinkles on forehead
• Lowered eyebrows
• Lowered corners of mouth

When the app detects these facial changes, it should label the expression "Disgust".