A HYBRID COMPUTER VISION-BASED SYSTEM FOR EXERCISE RECOGNITION, REP COUNTING, AND CALORIE BURN ESTIMATION
DOI:
https://doi.org/10.47526/3135-6877.193Keywords:
exercise recognition, repetition counting, calorie expenditure estimation, posture analysis, computer vision, human activity recognition, physical activity monitoringAbstract
Accurate exercise monitoring is essential for home workouts, physical education classes, and remote training support. However, manual rep counting is often unreliable, and calorie estimates are often simplified to generic formulas that fail to capture individual and activity-specific variations. This study proposes a hybrid computer vision-based system for exercise recognition, rep counting, and calorie burn estimation from monocular video. The proposed system combines marker-free pose extraction, landmark-based exercise state classification, exercise-aware rep counting, and a machine-learning-based calorie calculation model that incorporates user and activity descriptors. The proposed algorithm extracts body landmarks from video frames and transforms them into structured geometric features that support exercise recognition. The number of repetitions is then estimated from smoothed joint angle trajectories using motion-specific peak analysis, and calorie expenditure is determined using a calibrated regression model combined with motion-based intensity adjustment. The final exercise estimation model achieved an accuracy of 0.8886, a balanced accuracy of 0.8950, and a macro-F1 score of 0.8929. The calorie calculation model achieved a mean absolute error of 2.1339, a root mean square error of 3.6202, and an R² value of 0.9966. In an end-to-end self-test of the video recording, representative push-up and squat segments were correctly recognized, the number of repetitions was plausible, and calorie estimates remained within realistic short-term ranges. These results demonstrate that the proposed hybrid framework can provide an interpretable and practical solution for camera-based exercise monitoring without the need for dedicated wearable equipment or motion capture devices.
References
Ainsworth, B. E., Haskell, W. L., Herrmann, S. D., Meckes, N., Bassett, D. R., Jr., Tudor-Locke, C., Greer, J. L., Vezina, J., Whitt-Glover, M. C., & Leon, A. S. (2011). 2011 compendium of physical activities: A second update of codes and MET values. Medicine & Science in Sports & Exercise, 43(8), 1575–1581. https://doi.org/10.1249/MSS.0b013e31821ece12
Badiola-Bengoa, A., & Mendez-Zorrilla, A. (2021). A systematic review of the application of camera-based human pose estimation in the field of sport and physical exercise. Sensors, 21(18), 5996. https://doi.org/10.3390/s21185996
Bazarevsky, V., Grishchenko, I., Raveendran, K., Zhu, T., Zhang, F., & Grundmann, M. (2020). BlazePose: On-device real-time body pose tracking. arXiv. https://doi.org/10.48550/arXiv.2006.10204
Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 327(8476), 307–310. https://doi.org/10.1016/S0140-6736(86)90837-8
Brodersen, K. H., Ong, C. S., Stephan, K. E., & Buhmann, J. M. (2010). The balanced accuracy and its posterior distribution. In Proceedings of the 20th International Conference on Pattern Recognition (pp. 3121–3124). IEEE. https://doi.org/10.1109/ICPR.2010.764
Chicco, D., & Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21, 6. https://doi.org/10.1186/s12864-019-6413-7
Cornman, H. L., Stenum, J., Cowley, M., Zhang, Y., & Roemmich, R. T. (2021). Video-based quantification of human movement frequency using pose estimation: A pilot study. PLOS ONE, 16(12), e0261450. https://doi.org/10.1371/journal.pone.0261450
Erramchetty, S. K., Selvam, W. P., & Veerachamy, R. (2024). Human pose estimation using BlazePose. AIP Conference Proceedings, 2971(1), 040049. https://doi.org/10.1063/5.0196455
Hancock, J. T., & Khoshgoftaar, T. M. (2020). CatBoost for big data: An interdisciplinary review. Journal of Big Data, 7, 94. https://doi.org/10.1186/s40537-020-00369-8
Hussain, A., Zafar, K., Baig, A. R., Almakki, R., AlSuwaidan, L., & Khan, S. (2022). Sensor-based gym physical exercise recognition: Data acquisition and experiments. Sensors, 22(7), 2489. https://doi.org/10.3390/s22072489
Kaggle. (n.d.). Workout/exercises pose-landmark dataset [Data set]. Kaggle. https://www.kaggle.com/datasets
Kaggle. (2025). Predict calorie expenditure [Data set]. Kaggle Playground Series, Season 5, Episode 5. https://www.kaggle.com/competitions/playground-series-s5e5
Leone, A., Caroppo, A., Rescio, G., Manni, A., Siciliano, P., & Diraco, G. (2022). Ambient and wearable sensor technologies for energy expenditure quantification of ageing adults. Sensors, 22(13), 4893. https://doi.org/10.3390/s22134893
Lin, B.-S., Wang, L.-Y., Hwang, Y.-T., Chiang, P.-Y., & Chou, W.-J. (2021). Depth-camera based energy expenditure estimation system for physical activity using posture classification algorithm. Sensors, 21(12), 4216. https://doi.org/10.3390/s21124216
Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.-L., Yong, M. G., Lee, J., Chang, W.-T., Hua, W., Georg, M., & Grundmann, M. (2019). MediaPipe: A framework for building perception pipelines. arXiv. https://doi.org/10.48550/arXiv.1906.08172
Mercadal-Baudart, C., Liu, C.-J., Farrell, G., Boyne, M., González Escribano, J., Smolic, A., & Simms, C. (2024). Exercise quantification from single camera view markerless 3D pose estimation. Heliyon, 10(6), e27596. https://doi.org/10.1016/j.heliyon.2024.e27596
Nathan, D., Klasnja, P., Vankipuram, M., Momboisse, T., Powell, C., & Raju, D. (2015). Estimating physical activity energy expenditure with the Kinect sensor in an exergaming environment. PLOS ONE, 10(5), e0127113. https://doi.org/10.1371/journal.pone.0127113
Nguyen, H.-C., Dang, D.-T., Ngo, T.-D., & Pham, D.-T. (2023). Deep learning for human activity recognition on 3D human skeleton: Survey and comparative study. Sensors, 23(11), 5121. https://doi.org/10.3390/s23115121
Perrett, T., Masullo, A., Damen, D., Burghardt, T., Craddock, I., & Mirmehdi, M. (2022). Personalized energy expenditure estimation: Visual sensing approach with deep learning. JMIR Formative Research, 6(9), e33606. https://doi.org/10.2196/33606
Postlmayr, A., Garg, B., Cosman, P., & Dey, S. (2024). PersonalPT: One-shot approach for skeletal-based repetitive action counting for physical therapy. Smart Health, 34, 100516. https://doi.org/10.1016/j.smhl.2024.100516
Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). CatBoost: Unbiased boosting with categorical features. In Advances in Neural Information Processing Systems 31 (pp. 6638–6648). Curran Associates. https://doi.org/10.48550/arXiv.1706.09516
Roggio, F., Trovato, B., Sortino, M., & Musumeci, G. (2024). A comprehensive analysis of the machine learning pose estimation models used in human movement and posture analyses: A narrative review. Heliyon, 10(21), e39977. https://doi.org/10.1016/j.heliyon.2024.e39977
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Yassawi Journal of Engineering Science
This work is licensed under a Creative Commons Attribution 4.0 International License.
