Can two sources of movement data work better together?
A body pose describes how someone is positioned. Wearable inertial measurement units (IMUs) record acceleration and rotation over time. My research question was whether combining these signals could recognise fitness activities more effectively than using either one alone.
I used the existing MM-Fit dataset, covering 21 recording sessions, to implement a fusion model and compare it with pose-only and IMU-only models.

What I owned
This was my individual dissertation. I developed the data-processing and modelling workflow in Python and PyTorch, prepared the experimental inputs and evaluated the classifiers.
- Data preparation: I aligned pose and IMU sequences, prepared labelled windows and handled differences in scale and input structure.
- Model implementation: I built the pose CNN, the inertial LSTM and the classifier combining their learned features.
- Experimental comparison: I evaluated fusion against the single-modality models using classification metrics and confusion matrices.
- Technical interpretation: I examined generalisation and wearable-inference constraints, and documented what the experiments could support.
Different signals, one activity interval
The first challenge was alignment. Pose and inertial recordings have different sampling rates and structures. Combining unrelated windows would make the fusion experiment difficult to interpret, even if both branches trained successfully.
I prepared synchronised windows so the two inputs described the same activity interval. Pose windows contained 150 frames with 18 joints and three coordinates per joint; inertial windows contained 250 samples across 24 channels.


These figures show the two input types used in the research. Select a figure to view it at full size.
I also needed to handle missing pose observations, differences in input scale and class imbalance. My preprocessing used pose interpolation, hip-centred pose normalisation and inertial-channel standardisation. For training, I used non-activity downsampling, weighted cross-entropy and label smoothing.
These decisions connected the data preparation to the modelling problem: preserving movement information while limiting the influence of scale differences and dominant classes.
The pipeline I implemented

- Prepare paired examples. I aligned the modalities, normalised their inputs and created labelled activity windows.
- Learn from each modality. I used a CNN for the pose representation and an LSTM for temporal patterns in the inertial sequences.
- Fuse the features. I concatenated the two learned representations and used a fully connected classifier to predict the activity.
- Compare and inspect. I evaluated the fused model alongside the pose-only and IMU-only models, inspecting accuracy, precision, recall, F1 scores and confusion matrices.
I used NumPy, SciPy and scikit-learn alongside PyTorch for data processing and evaluation.
What the experiments showed
The comparison reported in my dissertation favoured the fused model over the pose-only and IMU-only models. Within those experiments, combining posture and inertial movement information was more effective than relying on a single modality.
Performance also varied between the evaluation groups. That made generalisation an important part of interpreting the result: strong performance on one set of recordings does not establish reliable performance for new users or environments.
The outcome was an implemented and evaluated multimodal workflow, with comparisons that made the contribution of each input visible. This finding concerns my experiments on MM-Fit. Results from the original researchers provided benchmark context and are separate from my own results.
What this work developed
I strengthened my ability to take heterogeneous time-series data through preprocessing, model implementation and evaluation. The project required me to connect decisions about windows, normalisation and class balance to the behaviour of the resulting classifiers.
I also developed a more careful approach to reporting model performance. Class-level precision, recall and macro F1 help explain behaviour that an overall accuracy figure can hide, especially when activities are unevenly represented.
What I would improve next
- Cross-user evaluation: I would make session and participant splits explicit in a saved experiment manifest, then repeat comparisons across held-out users.
- Missing-modality testing: I would test sensor disconnection and unreliable pose observations to understand how the fusion model behaves when an input deteriorates.
- Physical-device measurement: I would measure latency, memory use and energy consumption on the intended device. The project did not establish these through a physical wearable deployment.
The study focused on structured fitness activity. Exercise monitoring and rehabilitation are possible directions for further investigation; the project was not a clinically evaluated healthcare system.
Project context
Individual MSc Artificial Intelligence dissertation, University of Plymouth, September 2025. Research on CNN-LSTM fusion of 3D pose and IMU data using MM-Fit.
The full academic report remains private. This case study presents my implementation, experimental findings and development priorities.