SilentSpeak

Can a webcam turn a hand gesture into useful interface feedback?

I wanted to see what happens when a small computer-vision model has to live inside an actual product. Training a classifier was only the first part; the harder part was getting webcam input, hand landmarks, inference, confidence handling, and the interface to work together in real time.

My contribution

• Worked on the gesture-classification pipeline.
• Connected MediaPipe hand landmarks to browser inference.
• Integrated the classifier into the web interface with confidence feedback and sentence assembly.
• Worked on application integration and the separate FastAPI text-to-sign workflow.

Problem & approach

I worked on a gesture classifier using hand landmarks from MediaPipe and connected it to a web interface as part of a 36-hour team hackathon. Instead of stopping at a notebook result, I worked through the path from camera input to a prediction the user could see and use. The browser runs the trained Keras model through TensorFlow.js; a separate FastAPI service handles text-to-sign generation.

How it works

1. 01

   Capture

   Capture the user’s hand from the webcam.

2. 02

   Track landmarks

   Use MediaPipe to extract hand landmarks.

3. 03

   Normalise

   Transform the landmarks into the format expected by the classifier.

4. 04

   Predict

   Run the trained model and obtain a class/confidence prediction.

5. 05

   Respond

   Surface that prediction in the web interface.

Data & evaluation

Data & inputs

A limited 37-class landmark dataset. The frontend uses the same coordinate representation expected by the classifier.

How it was evaluated

The training notebook records 93.41% held-out test accuracy, rounded to 93.4%. This measures classification on the test samples, rather than unrestricted live sign-language interpretation.

Results & progress

93.4%

Test Accuracy

37

ASL Gestures

6,600+

Training Samples

A label and confidence from 37 gesture classes

• 37 gesture classes.
• Approximately 93.4% test accuracy on the project evaluation set.
• Working browser-based prototype.

Screenshots

View full size

Limits & lessons

Limitations

• SilentSpeak is a prototype, not a clinically validated assistive product. Test accuracy describes the project dataset and does not establish real-world accessibility performance.

What I learned

• This project pushed me toward ML engineering. A model becomes much more interesting once it has latency, interface, confidence, and integration constraints around it. I also learned that a good offline metric does not automatically mean the live experience feels good.

Next steps

• Test robustness under different lighting, camera angles, and users.
• Improve confidence handling and out-of-distribution behaviour.
• Reduce friction in the live interaction flow.