AccessKit · Michael Li

Background

For someone who cannot use a mouse or a keyboard, the face often keeps its range of movement. AccessKit asks how far a webcam alone can go as the input device.

Method

Head yaw and pitch come from OpenCV’s solvePnP on six facial landmarks. Blinks and winks come from each eye’s aspect ratio, mouth open and smile from ratios of the mouth’s landmarks, and a raised brow from its height above the eye. A state machine asks for most gestures to be held, for 0.5 to 1.5 seconds in the reader and 2.0 to 5.0 in the music controller, and waits out a cooldown between actions. Every mapping and threshold can be changed in a settings panel and is saved between sessions.

The reader finds the face with MediaPipe’s mesh of 478 landmarks and renders pages with PyMuPDF. The music controller uses dlib’s 68 landmarks, confirms a blink over three consecutive frames, drives Spotify through AppleScript, and also accepts a two-gesture sequence: one gesture held, then a second.

The ten gestures, how each is detected, and what each does by default. Every row can be remapped.
Gesture	Detected by	PDF reader	Music controller
Look right	Head turned right past 30° (reader) or 45° (music)	Next page	Volume up
Look left	Head turned left past 30° or 45°	Previous page	Play or pause
Look up	Head tilted up past 20° or 25°	Scroll up	Unassigned
Look down	Head tilted down past 20° or 25°	Scroll down	Unassigned
Smile	Mouth corners lifted past a set threshold	Zoom in	Held, then look right: volume up
Open mouth	Mouth aspect ratio above a set threshold	Reset zoom	Held, then look left: previous song
Raise eyebrows	Brow height over eye width (reader); brow height above the eyes, in pixels (music)	Zoom out	Unassigned
Long blink	Both eyes closed and held	Full screen on or off	Next song
Right wink	Right eye closed, left open	Unassigned	Unassigned
Left wink	Left eye closed, right open	Unassigned	Unassigned

From the PDF reader’s code (AccessibilityPDF.py), on MediaPipe’s canonical face model (Apache-2.0), whose 468 points are the first 468 of the 478 the reader’s mesh gives, in the same order; the ten iris points are not in the model.
Head pose: solvePnP fits the six landmarks named in A to the code’s six model points, divided by 4.5, drawn in D and E. Turned past 30° of yaw or 20° of pitch about the nose tip, the head is looking aside, up or down.
Eyes: (a + b) / 2c on each eye, the code’s (A + B) / 2C; B draws the eye with 33, and the other mirrors it. Below 0.25 on both is a blink, held 1.5 s a long blink; on one eye, a wink.
Brows: raised when the mean of the two brows’ heights e, over the eye’s width c, is above 0.85.
Mouth: open when v / h is above 0.6; a smile when the corners’ lift above the centre line s, over h, is above 0.02.
The music controller uses dlib’s 68 points; its thresholds are in the table.

Results

The table is the result: ten gestures, each found in the code with the rule shown and each remappable. No accuracy, latency or frame rate is given, because none was measured.

Limitations

Neither app has been tried by people with motor impairments, the people it was built for, so whether the gestures are comfortable or reliable for them is unknown. The remappable thresholds are the means of adapting it, not evidence that it adapts.

The defaults were tuned on one face, so they are a starting point that each user would need to adjust. The two apps use different landmark models and share no code, and the music controller measures a raised brow in pixels, so its threshold shifts with distance from the camera; the reader’s ratio does not.