Skip to main content
SpringerLink
Log in

A deep learning model for ergonomics risk assessment and sports and health monitoring in self-occluded images

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Ergonomic assessments and sports and health monitoring play a crucial role and have contributed to sustainable development in many areas such as product architecture, design, health, and safety as well as workplace design. Recently, visual ergonomic assessments have been broadly employed for skeleton analysis of human joints for body postures localization and classification to deal with musculoskeletal disorders risks. Moreover, monitoring players in a sports activity helps to analyze their actions to help maximize body performance. However, body postures identification has some limitations in self-occlusion joint postures. In this study, a visual ergonomic assessment technique employing a multi-frame and multi-path convolutional neural network (CNN) is presented to assess ergonomic risks in the presence of free-occlusion and self-occlusion conditions. Our model has four inputs that accept four sequential frames to overcome the problems of the missing joints and classify the input into one of four risk categories. Our pipeline was evaluated on a video with 5 min ~ 300 s (that could be 9000 frames) duration time and showed that our architecture has competitive results (recall = 0.8925, precision = 0.8743, F-score = 0.8837).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Availability of data and materials

Data are available on request due to privacy/ethical restrictions.

References

  1. Radjiyev, A., Qiu, H., Xiong, S., Nam, K.H.: Ergonomics and sustainable development in the past two decades (1992–2011): research trends and how ergonomics can contribute to sustainable development. Appl. Ergon. 46, 67–75 (2015). https://doi.org/10.1016/J.APERGO.2014.07.006

    Article  PubMed  Google Scholar 

  2. Zelik, K.E., Nurse, C.A., Schall, M.C., Sesek, R.F., Marino, M.C., Gallagher, S.: An ergonomic assessment tool for evaluating the effect of back exoskeletons on injury risk. Appl. Ergon. 99, 103619 (2022). https://doi.org/10.1016/J.APERGO.2021.103619

    Article  PubMed  Google Scholar 

  3. Plantard, P., Shum, H.P.H., Le Pierres, A.S., Multon, F.: Validation of an ergonomic assessment method using Kinect data in real workplace conditions. Appl. Ergon. 65, 562–569 (2017). https://doi.org/10.1016/J.APERGO.2016.10.015

    Article  PubMed  Google Scholar 

  4. Kee, D.: Comparison of OWAS, RULA and REBA for assessing potential work-related musculoskeletal disorders. Int. J. Ind. Ergon. 83, 103140 (2021). https://doi.org/10.1016/J.ERGON.2021.103140

    Article  Google Scholar 

  5. Ramaganesh, M., Jayasuriyan, R., Rajpradeesh, T., Bathrinath, S., Manikandan, R.: Ergonomics hazard analysis techniques—a technical review. Mater. Today Proc. 46, 7789–7797 (2021). https://doi.org/10.1016/J.MATPR.2021.02.329

    Article  Google Scholar 

  6. Li, X.: A visual ergonomic assessment approach using Kinect and OWAS in real workplace environments. Multiscale Multidiscip. Model. Exper. Des. (2022). https://doi.org/10.1007/S41939-022-00133-W/TABLES/9

    Article  Google Scholar 

  7. Iqbal, M., Angriani, L., Hasanuddin, I., Erwan, F., Soewardi, H., Hassan, A.: Working posture analysis of wall building activities in construction works using the OWAS method. IOP Conf. Ser. Mater. Sci. Eng. 1082(1), 012008 (2021). https://doi.org/10.1088/1757-899X/1082/1/012008

    Article  Google Scholar 

  8. MassirisFernández, M., Fernández, J.Á., Bajo, J.M., Delrieux, C.A.: Ergonomic risk assessment based on computer vision and machine learning. Comput. Ind. Eng. 149, 106816 (2020). https://doi.org/10.1016/J.CIE.2020.106816

    Article  Google Scholar 

  9. Kee, D.: Development and evaluation of the novel postural loading on the entire body assessment. Ergonomics (2021). https://doi.org/10.1080/00140139.2021.1903084

    Article  PubMed  Google Scholar 

  10. Otto, M., Lampen, E., Auris, F., Gaisbauer, F., Rukzio, E.: Applicability evaluation of kinect for EAWS ergonomic assessments. Procedia CIRP 81, 781–784 (2019). https://doi.org/10.1016/J.PROCIR.2019.03.194

    Article  Google Scholar 

  11. Kalkis, H., Graveris, I., Roja, Z.: Ergonomic indicators and physical workload risks in food production and possibilities for risk prevention. Lecture Notes Netw. Syst. 273, 47–53 (2021). https://doi.org/10.1007/978-3-030-80713-9_7

    Article  Google Scholar 

  12. Lun, R., Zhao, W.: A survey of applications and human motion recognition with microsoft kinect. Int. J. Pattern Recognit. Artif. Intell. (2015). https://doi.org/10.1142/S0218001415550083

    Article  Google Scholar 

  13. Zennaro, S., et al.: Performance evaluation of the 1st and 2nd generation Kinect for multimedia applications. Proc. (IEEE Int. Conf. Multimed. Expo.) (2015). https://doi.org/10.1109/ICME.2015.7177380

    Article  Google Scholar 

  14. Wang, Q., Kurillo, G., Ofli, F., Bajcsy, R.: Evaluation of pose tracking accuracy in the first and second generations of microsoft Kinect. In: Proceedings—2015 IEEE International Conference on Healthcare Informatics, ICHI 2015, pp. 380–389, (2015). https://doi.org/10.1109/ICHI.2015.54.

  15. Müller, B., Ilg, W., Giese, M.A., Ludolph, N.: Validation of enhanced kinect sensor based motion capturing for gait assessment. PLoS ONE 12(4), e0175813 (2017). https://doi.org/10.1371/JOURNAL.PONE.0175813

    Article  PubMed  PubMed Central  Google Scholar 

  16. Nour, M., Cömert, Z., Polat, K.: A novel medical diagnosis model for COVID-19 infection detection based on deep features and bayesian optimization. Appl. Soft Comput. J. (2020). https://doi.org/10.1016/j.asoc.2020.106580

    Article  Google Scholar 

  17. Baghban, A., Kashiwao, T., Bahadori, M., Ahmad, Z., Bahadori, A.: Estimation of natural gases water content using adaptive neuro-fuzzy inference system. Petrol. Sci. Technol. 34(10), 891–897 (2016). https://doi.org/10.1080/10916466.2016.1176039

    Article  CAS  Google Scholar 

  18. Ahmadi, M.H., Baghban, A., Ghazvini, M., Hadipoor, M., Ghasempour, R., Nazemzadegan, M.R.: An insight into the prediction of TiO2/water nanofluid viscosity through intelligence schemes. J. Therm. Anal. Calorim. 139(3), 2381–2394 (2020). https://doi.org/10.1007/S10973-019-08636-4/FIGURES/11

    Article  CAS  Google Scholar 

  19. Fardad, M., Muntean, G.-M., Tal, I.: Latency-aware V2X operation mode coordination in vehicular network slicing. In: 2023 IEEE 97th Vehicular Technology Conference (VTC2023-Spring). (2023). https://doi.org/10.1109/VTC2023-SPRING57618.2023.10200069

  20. Fardad, M., Mianji, E.M., Muntean, G.M., Tal, I.: A fast and effective graph-based resource allocation and power control scheme in vehicular network slicing. In: IEEE International Symposium on Broadband Multimedia Systems and Broadcasting, BMSB. (2022). https://doi.org/10.1109/BMSB55706.2022.9828750

  21. Mianji, E.M., Muntean, G.M., Tal, I.: Trustworthy routing in VANET: a Q-learning approach to protect against black hole and gray hole attacks. In: 2023 IEEE 97th Vehicular Technology Conference (VTC2023-Spring). (2023). https://doi.org/10.1109/VTC2023-SPRING57618.2023.10201086

  22. Lui, H.W., Chow, K.L.: Multiclass classification of myocardial infarction with convolutional and recurrent neural networks for portable ECG devices. Inf. Med. Unlock. 13, 26–33 (2018). https://doi.org/10.1016/j.imu.2018.08.002

    Article  Google Scholar 

  23. Girdhar, R., Gkioxari, G., Torresani, L., Paluri, M., Tran, D.: Detect-and-track: efficient pose estimation in videos. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 350–359, 2017. https://doi.org/10.1109/CVPR.2018.00044

  24. Aiman, A., Shen, Y., Bendechache, M., Inayat, I., Kumar, T.: AUDD: audio Urdu digits dataset for automatic audio Urdu digit recognition. Appl. Sci. 11(19), 8842 (2021). https://doi.org/10.3390/APP11198842

    Article  Google Scholar 

  25. Aleem, S., Kumar, T., Little, S., Bendechache, M., Brennan, R., McGuinness, K.: Random Data Augmentation based Enhancement: AGeneralized Enhancement Approach for Medical Datasets. (2021)

  26. Tataei Sarshar, N. et al.: Glioma brain tumor segmentation in four mri modalities using a convolutional neural network and based on a transfer learning method. pp. 386–402. (2023). https://doi.org/10.1007/978-3-031-04435-9_39

  27. eve Chiasson, M., Imbeau, D., Major, J., Aubry, K., Delisle, A.: Influence of musculoskeletal pain on workers’ ergonomic risk-factor assessments. Appl. Ergon. 49, 1–7 (2015). https://doi.org/10.1016/J.APERGO.2014.12.011

    Article  PubMed  Google Scholar 

  28. Brandl, C., Mertens, A., Schlick, C.M.: Effect of sampling interval on the reliability of ergonomic analysis using the Ovako working posture analysing system (OWAS). Int. J. Ind. Ergon. 57, 68–73 (2017). https://doi.org/10.1016/J.ERGON.2016.11.013

    Article  Google Scholar 

  29. Shotton, J., et al.: Real-time human pose recognition in parts from single depth images. Commun. ACM 56(1), 116–124 (2013). https://doi.org/10.1145/2398356.2398381

    Article  Google Scholar 

  30. Park, Y., Moon, S., Suh, I.H.: Tracking human-like natural motion using deep recurrent neural networks. (2016)

  31. Parsa, B., Banerjee, A.G.: A multi-task learning approach for human activity segmentation and ergonomics risk assessment. pp. 2352–2362. (2021)

  32. Zhang, H., Yan, X., Li, H.: Ergonomic posture recognition using 3D view-invariant features from single ordinary camera. Autom. Constr. 94, 1–10 (2018). https://doi.org/10.1016/J.AUTCON.2018.05.033

    Article  CAS  Google Scholar 

  33. Lunin, A., Glock, C.H.: Systematic review of Kinect-based solutions for physical risk assessment in manual materials handling in industrial and laboratory environments. Comput. Ind. Eng. 162, 107660 (2021). https://doi.org/10.1016/J.CIE.2021.107660

    Article  Google Scholar 

  34. Battini, D., Berti, N., Finco, S., Guidolin, M., Reggiani, M., Tagliapietra, L.: WEM-Platform: a real-time platform for full-body ergonomic assessment and feedback in manufacturing and logistics systems. Comput. Ind. Eng. 164, 107881 (2022). https://doi.org/10.1016/J.CIE.2021.107881

    Article  Google Scholar 

  35. Humadi, A., Nazarahari, M., Ahmad, R., Rouhani, H.: In-field instrumented ergonomic risk assessment: inertial measurement units versus Kinect V2. Int. J. Ind. Ergon. 84, 103147 (2021). https://doi.org/10.1016/J.ERGON.2021.103147

    Article  Google Scholar 

  36. Zambrano Moya, L., Baydal-Bertomeú, J.M., Baño Morales, D., Fuentes Rosero, P., Zambrano Orejuela, I., Cesén Arteaga, M.: Ergonomic study on nurses that attend the feeding task to neonates through data acquisition, validation, and processing obtained from depth sensors. Mater. Today Proc. (2021). https://doi.org/10.1016/J.MATPR.2021.07.436

    Article  Google Scholar 

  37. Diego-Mas, J.A., Alcaide-Marzal, J.: Using kinect™ sensor in observational methods for assessing postures at work. Appl. Ergon. 45(4), 976–985 (2014). https://doi.org/10.1016/J.APERGO.2013.12.001

    Article  PubMed  Google Scholar 

  38. Mahmood, A. et al.: Deep Learning for Coral Classification. In: Handbook of Neural Computation, Elsevier Inc., pp. 383–401. (2017). https://doi.org/10.1016/B978-0-12-811318-9.00021-1

  39. Anari, S., Tataei Sarshar, N., Mahjoori, N., Dorosti, S., Rezaie, A.: Review of deep learning approaches for thyroid cancer diagnosis. Math. Probl. Eng. (2022). https://doi.org/10.1155/2022/5052435

    Article  Google Scholar 

  40. Ranjbarzadeh, R., Caputo, A., Tirkolaee, E.B., Jafarzadeh Ghoushchi, S., Bendechache, M.: Brain tumor segmentation of MRI images: a comprehensive review on the application of artificial intelligence tools. Comput. Biol. Med. 152(106405), 2023 (2023). https://doi.org/10.1016/J.COMPBIOMED.2022.106405

    Article  Google Scholar 

  41. Ranjbarzadeh, R., et al.: A deep learning approach for robust, multi-oriented, and curved text detection. Cognit. Comput. 1, 1–13 (2022). https://doi.org/10.1007/S12559-022-10072-W/FIGURES/10

    Article  Google Scholar 

  42. Ranjbarzadeh, R., et al.: Breast tumor localization and segmentation using machine learning techniques: overview of datasets, findings, and methods. Comput. Biol. Med. 152, 106443 (2023). https://doi.org/10.1016/J.COMPBIOMED.2022.106443

    Article  PubMed  Google Scholar 

  43. Safavi, S., Jalali, M.: RecPOID: POI recommendation with friendship aware and deep CNN. Future Internet 13(3), 79 (2021). https://doi.org/10.3390/FI13030079

    Article  Google Scholar 

  44. Safavi, S., Jalali, M.: DeePOF: a hybrid approach of deep convolutional neural network and friendship to Point-of-Interest (POI) recommendation system in location-based social networks. Concurr. Comput. 34(15), e6981 (2022). https://doi.org/10.1002/CPE.6981

    Article  Google Scholar 

  45. Akhtar, N., Ragavendran, U.: Interpretation of intelligence in CNN-pooling processes: a methodological survey. Neural Comput. Appl. 32(3), 879–898 (2020). https://doi.org/10.1007/s00521-019-04296-5

    Article  Google Scholar 

  46. Wang, S.H., Lv, Y.D., Sui, Y., Liu, S., Wang, S.J., Zhang, Y.D.: Alcoholism detection by data augmentation and convolutional neural network with stochastic pooling. J. Med. Syst. 42(1), 1–11 (2017). https://doi.org/10.1007/S10916-017-0845-X

    Article  Google Scholar 

  47. Kimizuka, M., Kim, S., Yamashita, M.: Solving pooling problems with time discretization by LP and SOCP relaxations and rescheduling methods. J. Global Optim. 75(3), 631–654 (2019). https://doi.org/10.1007/S10898-019-00795-W

    Article  MathSciNet  Google Scholar 

  48. Zhou, Q., Qu, Z., Cao, C.: Mixed pooling and richer attention feature fusion for crack detection. Pattern Recognit. Lett. 145, 96–102 (2021). https://doi.org/10.1016/J.PATREC.2021.02.005

    Article  ADS  Google Scholar 

  49. Ning, X., Tian, W., Yu, Z., Li, W., Bai, X., Wang, Y.: HCFNN: High-order coverage function neural network for image classification. Pattern Recognit. 131, 108873 (2022). https://doi.org/10.1016/J.PATCOG.2022.108873

    Article  Google Scholar 

  50. Liang, G., Kintak, U., Ning, X., Tiwari, P., Nowaczyk, S., Kumar, N.: Semantics-aware dynamic graph convolutional network for traffic flow forecasting. IEEE Trans. Veh. Technol. (2023). https://doi.org/10.1109/TVT.2023.3239054

    Article  Google Scholar 

  51. Kasgari, A.B., Safavi, S., Nouri, M., Hou, J., Sarshar, N.T., Ranjbarzadeh, R.: Point-of-interest preference model using an attention mechanism in a convolutional neural network. Bioengineering 10(4), 495 (2023). https://doi.org/10.3390/BIOENGINEERING10040495

    Article  PubMed  PubMed Central  Google Scholar 

  52. Ning, X., Gou, D., Dong, X., Tian, W., Yu, L., Wang, C.: Conditional generative adversarial networks based on the principle of homologycontinuity for face aging. Concurr. Comput. 34(12), e5792 (2022). https://doi.org/10.1002/CPE.5792

    Article  Google Scholar 

  53. Cai, W., et al.: A novel hyperspectral image classification model using bole convolution with three-directions attention mechanism: small sample and unbalanced learning. IEEE Trans. Geosci. Remote Sens. (2022). https://doi.org/10.1109/TGRS.2022.3201056

    Article  Google Scholar 

  54. Ranjbarzadeh, R., et al.: ME-CCNN: multi-encoded images and a cascade convolutional neural network for breast tumor segmentation and recognition. Artif. Intell. Rev. 2023, 1–38 (2023). https://doi.org/10.1007/S10462-023-10426-2

    Article  MathSciNet  Google Scholar 

  55. Hoy, J., et al.: Whole body vibration and posture as risk factors for low back pain among forklift truck drivers. J. Sound Vib. 284(3–5), 933–946 (2005). https://doi.org/10.1016/J.JSV.2004.07.020

    Article  ADS  Google Scholar 

  56. Chowdhury Salian, S., Boricha, J., Yardi, S.: Identification of awkward postures that cause discomfort to liquid petroleum gas workers in Mumbai, India. Indian J. Occup. Environ. Med. 16(1), 3 (2012). https://doi.org/10.4103/0019-5278.99679

    Article  PubMed  PubMed Central  Google Scholar 

  57. Mattila, M., Karwowski, W., Vilkki, M.: Analysis of working postures in hammering tasks on building construction sites using the computerized OWAS method. Appl. Ergon. 24(6), 405–412 (1993). https://doi.org/10.1016/0003-6870(93)90172-6

    Article  CAS  PubMed  Google Scholar 

  58. Mousavi, S.M., Asgharzadeh-Bonab, A., Ranjbarzadeh, R.: Time-frequency analysis of EEG signals and GLCM features for depth of anesthesia monitoring. Comput. Intell. Neurosci. 2021, 1–14 (2021). https://doi.org/10.1155/2021/8430565

    Article  CAS  Google Scholar 

  59. Ranjbarzadeh, R., et al.: A deep learning approach for robust, multi-oriented, and curved text detection. Cognit. Comput. 1, 1–13 (2022). https://doi.org/10.1007/S12559-022-10072-W

    Article  Google Scholar 

  60. Ranjbarzadeh, R., Zarbakhsh, P., Caputo, A., Tirkolaee, E.B., Bendechache, M.: Brain tumor segmentation based on an optimized convolutional neural network and an improved chimp optimization algorithm. (2022). https://doi.org/10.21203/RS.3.RS-2203596/V1

Download references

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Author information

Authors and Affiliations

  1. School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran

    Amirhossein Aghamohammadi & Seyed Aliasghar Beheshti Shirazi

  2. Department of Electrical and Computer Engineering, Buien Zahra Technical University, Buien Zahra, Iran

    Seyed Yashar Banihashem & Saman Shishechi

  3. School of Computing, Faculty of Engineering and Computing, Dublin City University, Dublin, Ireland

    Ramin Ranjbarzadeh

  4. Faculty of Industrial Engineering, Urmia University of Technology, Urmia, Iran

    Saeid Jafarzadeh Ghoushchi

  5. Lero & ADAPT Research Centres, School of Computer Science, University of Galway, Galway, Ireland

    Malika Bendechache

Authors
  1. Amirhossein Aghamohammadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Seyed Aliasghar Beheshti Shirazi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Seyed Yashar Banihashem
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Saman Shishechi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ramin Ranjbarzadeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Saeid Jafarzadeh Ghoushchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Malika Bendechache
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

AA and SABS were involved in conceptualization, investigation, methodology, and software. SYB was involved in data preparation and curation, investigation, and writing the paper. SS was involved in data preparation and curation, formal analysis, and writing the paper. RR was a supervisor and involved in reviewing and editing and investigation. SJG was involved in Investigation, methodology, validation, and formal analysis. MB was a supervisor and involved in reviewing and editing and validation.

Corresponding author

Correspondence to Ramin Ranjbarzadeh.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethical approval

This study was approved by the ethics committee of … (reference no. 2021–062).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aghamohammadi, A., Beheshti Shirazi, S.A., Banihashem, S.Y. et al. A deep learning model for ergonomics risk assessment and sports and health monitoring in self-occluded images. SIViP 18, 1161–1173 (2024). https://doi.org/10.1007/s11760-023-02830-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02830-6

Keywords

Search

Navigation