Exercise Form Assessment from Video
With fifteen templates a 2007 gesture recogniser beat both classical ML and a CNN-LSTM. On a public benchmark with seven thousand samples the ranking reversed entirely. The finding is about data regime, not model sophistication.
An unpublished research exploration, paused before publication.
- Dataset
- A recorded boxing corpus of 15 template videos (five actions × three execution-quality levels) with an unseen 59-second test video containing 19 actions; benchmarked against the UCI Human Activity Recognition Using Smartphones dataset (7,352 train / 2,947 test).
- Evaluation
- One fixed unseen test video, predictions timestamp-aligned to ground truth, with the same test set applied identically to all three approaches.
Research questions
- Can execution quality — not just action identity — be recognised from pose landmarks alone?
- Which method family is appropriate when only one real example per class exists?
- Does that answer survive contact with a dataset that has thousands of examples per class?
The $1 Recognizer correctly classified 13 of 19 actions in the unseen 59-second test video. It beat both classical ML and the CNN-LSTM on this task.
LDA on the UCI Human Activity Recognition Using Smartphones benchmark — six everyday activities from phone accelerometer and gyroscope, 7,352 training samples. This is a public benchmark result, not a result on the exercise-form corpus.
Status
The work was paused before publication, so I present it here as an unpublished research exploration rather than as a completed paper.
Research question
Automated exercise-form feedback needs to recognise more than which action someone performed — it needs to recognise how well. That makes it a harder problem than standard activity recognition, because the classes are not "jab versus cross" but "jab performed well versus jab performed badly", and the visual difference between those is small.
The practical question underneath it turned out to be more interesting: when you have almost no data, which family of methods is actually appropriate?
Corpus
Five boxing actions — jab, cross, uppercut, left hook, right hook — each recorded at three execution-quality levels: perfect, moderate, and fail. Fifteen template videos in total, which is one real video per class.
Eight body landmarks were tracked per frame with MediaPipe: wrists, elbows, shoulders and hips.
The test set is a single unseen 59-second video containing 19 actions — nine performed perfectly, five moderately, five badly.
Method
Three approaches, all evaluated on that same unseen video.
The $1 Recognizer. A 2007 template-matching gesture algorithm, operating on raw wrist coordinates over a 30-frame sliding window. It is designed for exactly this situation: one example per gesture, no training.
Classical ML via PyCaret. Each action reduced to a single row of statistical features — position mean, variance, range; velocity and acceleration; elbow and shoulder joint angles; inter-hand distance. 143 features initially, later expanded to 359 by adding skewness, kurtosis, percentiles, IQR, energy, zero-crossing rate, FFT dominant frequency and angular velocity. Augmentation — Gaussian noise, random temporal sub-sampling, spatial scaling — expanded 15 real videos to 165 samples.
CNN-LSTM. Raw landmark sequences fed directly in, 16 values per frame across 60 frames, letting the network learn its own representation instead of using hand-crafted features.
Results on the recorded corpus
| Method | Correct | Accuracy |
|---|---|---|
| $1 Recognizer | 13/19 | 68.4% |
| Classical ML — Random Forest, 143 features | 11/19 | 57.9% |
| Classical ML — LDA, 359 features | 10/19 | 52.6% |
| CNN-LSTM | 8/19 | 42.1% |
By execution quality, the $1 Recognizer scored 5/9 on perfect actions, 4/5 on moderate, and 4/5 on fail.
The oldest and simplest method won by a wide margin.
Negative findings
These are the results worth keeping.
More features made it worse. Expanding from 143 to 359 features reduced the best classical accuracy from 57.9% to 52.6%. With 165 augmented samples drawn from 15 real ones, additional dimensions bought noise rather than signal — the curse of dimensionality behaving exactly as advertised.
The CNN-LSTM fit its training set completely. 100% training accuracy against 42.1% on the test video is the clearest possible demonstration of what fifteen real examples can and cannot support.
Augmentation did not rescue anything. Noise, temporal sub-sampling and spatial scaling generate variations of a single performance. They do not generate a second performance, and the models could not tell the difference.
Perfect execution was hardest to classify. The $1 Recognizer scored 55.6% on perfect actions against 80% on both moderate and failed ones. Cleanly executed punches were most often confused with each other; badly executed ones were more distinctive.
Segmentation and scoring
The sliding-window detector produced 34 detection segments for 19 actions, because an action spanning more than one 30-frame window triggers a classification in each.
I handled that by aligning predictions to ground-truth actions on timestamp, so the reported figures measure classification given correct segmentation. Treating segmentation and classification as two measurable stages rather than one blended score is what makes the comparison across the three approaches meaningful — every method is scored the same way, on the same segments.
The benchmark that reversed everything
To find out whether "template matching wins" was a real conclusion or an artefact of having almost no data, all three approaches were re-run on the UCI Human Activity Recognition Using Smartphones dataset: six everyday activities from phone accelerometer and gyroscope, with 7,352 training and 2,947 test samples.
| Method | UCI HAR | Boxing corpus |
|---|---|---|
| LDA | 96.6% | 52.6% |
| Logistic Regression | 96.2% | — |
| SVM (RBF) | 96.1% | — |
| CNN-LSTM | 91.3% | 42.1% |
| $1 Recognizer | 30.1% | 68.4% |
The ranking inverts completely. The method that won on 15 templates finishes last on 7,352 samples, and the methods that failed on 15 templates dominate.
Interpretation
This is the finding. Not 96.6%, and not 68.4% — the reversal.
Method selection in this problem is governed by data regime, not by model sophistication. Template matching encodes a strong prior ("this looks like that stored example") which is exactly right when you have one example and hopelessly rigid when you have thousands. Learned models encode almost no prior, which is useless at fifteen samples and decisive at seven thousand.
Reporting only the 96.6% would tell a much better-sounding and much less true story. It is a result on a public benchmark of walking, sitting and standing — a substantially easier task than distinguishing a well-thrown jab from a poorly-thrown one, on data that was not collected for this project.
Study design and scope
- One real video per class, which is precisely the regime the study set out to characterise, and the reason the reversal against UCI HAR is the interesting result.
- A single 59-second test video, held out and used identically across all three approaches.
- One performer, one environment, one camera angle, keeping the comparison controlled.
- Execution-quality labels assigned by the performer, which is what made three graded levels per action feasible to record at all.
- UCI HAR is a different modality — phone inertial sensors rather than pose landmarks — which is what makes it a genuinely independent check on the method comparison.
Where I would take this next
At least 50 genuine examples per class from multiple performers, which is the threshold where the learned approaches would have something to learn from. Multiple camera angles. Independently assigned quality labels, with inter-rater agreement reported. Segmentation measured as its own task with its own metric, rather than assumed away. And subject-level splits, so a model cannot succeed by recognising the person instead of the movement.
Acknowledgement
Supervised by Prof. Ayman Ezzat, Computer Science Programme Director at European Universities in Egypt (University of London branch campus) — ayman.ezzat@eue.edu.eg.
The direction of this study is his. The benchmark comparison that produced its most useful result came from his instruction to stop recording new video and test the same three approaches against a standard HAR dataset — which is what exposed the reversal.
Original artifacts
The full experimental record, written up below.
Notes on evidence
- An unpublished research exploration, paused before publication.
- Supervised by Prof. Ayman Ezzat and credited with permission.
- The 96.6% figure is the UCI HAR public-benchmark result; 68.4% is the result on the recorded boxing corpus.