Camera Coach: Activity Recognition and Assessment Using Thermal and RGB Videos

Authors

Ahmed Sharshar, Mohamed Bin Zayed University of Artificial IntelligenceFollow
Ahmed Hesham Aboeitta, Mohamed Bin Zayed University of Artificial IntelligenceFollow
Ahmed Fayez, Egypt-Japan University of Science and Technology
Mohamed A. Khamis, Egypt-Japan University of Science and Technology
Ahmed B. Zaky, Egypt-Japan University of Science and Technology
Walid Gomaa, Egypt-Japan University of Science and Technology

Document Type

Conference Proceeding

Publication Title

Proceedings of the International Joint Conference on Neural Networks

Abstract

Human Activity Recognition (HAR) has sparked a lot of attention because of its wide range of applications in sports, animation, simulation, and entertainment. There are many applications based on HAR, like healthcare monitoring, entertainment applications, and athletes' performance analysis. During the Corona pandemic, people took remote training as a safe and easy solution to do sports activities during the closure period, so there was a need to use smart systems to help people to train alone. In this paper, we expand our work on our dataset Multi-Modal Dataset of Sports (MMDOS); this dataset contains a variety of data forms from different sensors, including RGB videos, inertial motion data, depth, and thermal data, all synchronized with regard to the activities/exercises performed. The dataset consists of four workout exercises: free squats, shoulder press, push-ups, and lunges. The exercises are performed by 50 participants indoors at a fitness center. Professional trainers label the data with the type of activity and the participant's mistakes. Data are collected from different sensors, including Red-Green-Blue (RGB) videos, depth videos and skeleton data, inertial motion units (IMU) data, and thermal data; all are synchronized. From these modalities, we specifically used RGB, thermal videos, and 3-D skeleton of depth videos to recognize and asses the four activities. We built a smart Gym coaching model using classical machine learning (ML) and deep learning (DL) methods. The proposed system achieved an outstanding performance of 99% for activity recognition and an error of 0.75 out of 10 for activity assessment.

DOI

10.1109/IJCNN54540.2023.10191379

Publication Date

8-2-2023

Keywords

Artificial Gym Coaching, Deep Learning, Human Activity Recognition, Inertial Measurement Unit, Pose Estimation, Random Forest

Comments

IR conditions: non-described

Recommended Citation

A. Sharshar, A. H. A. Eitta, A. Fayez, M. A. Khamis, A. B. Zaky and W. Gomaa, "Camera Coach: Activity Recognition and Assessment Using Thermal and RGB Videos," 2023 International Joint Conference on Neural Networks (IJCNN), Gold Coast, Australia, 2023, pp. 1-8, doi: 10.1109/IJCNN54540.2023.10191379.

Additional Links

https://doi.org/10.1109/IJCNN54540.2023.10191379