The model was the easy part
Training the classifier was relatively straightforward. I used MediaPipe to turn hand positions into landmarks, then trained a model to recognise 37 gestures. Eventually, I got the test accuracy to around 93.4%.
That felt great for about five minutes.
The problem was that a model sitting inside a notebook is not particularly useful. SilentSpeak only became interesting when I started asking what had to happen between someone holding their hand in front of a camera and a prediction appearing on the screen. That gap ended up teaching me more than the classifier itself.
What the browser actually sees
The model does not see a hand. It sees numbers.
The webcam gives me an image. MediaPipe finds the hand and converts it into landmark coordinates. Those coordinates then need to be represented in the same way as the data the model was trained on.
That sounds obvious now, but it made me understand why consistency between training and inference matters so much. If the representation changes between the two, even slightly, a perfectly good model can suddenly look terrible.
Accuracy was only one part of it
A 93.4% test accuracy sounds like the important number. It was not the only thing that mattered once the model was running live.
Predictions could jump between classes. Hands moved in and out of frame. Some gestures looked similar. The camera conditions were never exactly the same as the training data.
That made the interface part of the ML problem too. Confidence thresholds, feedback, and how predictions were presented all affected whether the system actually felt usable. It was the first project that really made me see the difference between training a model and engineering something around one.
What I'd do differently
If I rebuilt SilentSpeak now, I would spend much more time collecting better data before touching the model architecture.
I would also evaluate gestures individually instead of treating the overall accuracy as the whole story. Some classes are naturally much harder to distinguish than others, and that is probably more useful to know than a single average number.
The biggest lesson is still the same one: getting a model to work is only half the job. Getting it to behave properly inside software is where things get interesting.
This note covers the project as described in my portfolio. The linked source and demo are the best places to look at the implementation.