SkyWrite
Real-Time Touchless AI Whiteboard & Character Recognition
SkyWrite is a computer vision-driven interface that transforms any standard webcam into an interactive touchless drawing canvas. By pairing sub-pixel hand landmark tracking with a deep convolutional neural network, users can draw characters in mid-air and have them recognized and digitized in real time.
CNN Accuracy
EMNIST ByClass Dataset
Hand Tracking
Real-Time MediaPipe
Rendering Pipeline
Ultra-Low Latency
Inference Time
Async Background Thread
Technology Stack
System Architecture & Pipeline
How data flows from camera sensor to neural network transcription:
Video Ingestion & 21-Point Landmark Tracking
OpenCV • MediaPipe HandsCaptures frames at 60 FPS via OpenCV and passes RGB buffers to Google MediaPipe Hands, extracting 21 normalized 3D keypoints per frame to pinpoint index fingertip trajectory with sub-pixel precision.
Stateful Gesture State Machine
Euclidean Distance • Vector MathComputes inter-finger angular vectors and euclidean distances to reliably classify hand states: Index extended = DRAW_MODE, Index + Middle = HOVER_MODE, Closed Fist = CLEAR_CANVAS.
Temporal Stroke Smoothing & Canvas Blending
Exponential Moving Average • NumPyEliminates webcam frame-to-frame noise and micro-tremors using an Exponential Moving Average (EMA) filter, rendering crisp anti-aliased Bézier paths over a dual-layer alpha canvas.
Contour Segmentation & Preprocessing
OpenCV Morphology • Bounding Box NormalizationWhen drawing pauses, the canvas extracts the active stroke bounding box, normalizes aspect ratios, centers the glyph with 4px padding, and resizes to 28x28 grayscale to match the EMNIST distribution.
Deep CNN Classification & Transcription
TensorFlow • Keras • EMNIST ByClassA convolutional neural network with Conv2D, BatchNormalization, Dropout, and Dense softmax layers predicts the drawn alphanumeric character with ~87.4% test accuracy in under 15ms.
Key Technical Challenges & Solutions
Non-trivial algorithmic hurdles solved during implementation:
Raw fingertip coordinates suffered from frame-by-frame sensor noise, leading to jagged, disconnected handwriting on screen.
Implemented a weighted temporal smoothing pipeline (EMA) that dynamically adjusts smoothing factors based on fingertip velocity — preserving sharp corners while eliminating resting jitter.
Natural hand rotations in 3D space frequently triggered unintended drawing strokes when users merely intended to gesture or pause.
Built a stateful thresholding system checking both fingertip elevation relative to the PIP joints and distance to the thumb MCP, preventing accidental pen-down triggers.
Running synchronous deep learning predictions on every frame caused significant frame drops and stuttered video feeds.
Decoupled canvas drawing from character classification using asynchronous non-blocking worker threads, sustaining smooth 60 FPS drawing alongside instant recognition.
Explore the Source Code
The full codebase including gesture state machines, training pipelines, and dataset scripts is open-sourced on GitHub.