Health & Sensing at Ubicomp'24
Highlights and Reflections from the 2024
In this article, we share reflections from IClab students who attended the 2025 Ubicomp/ISWS conference, held in Melbourne, Australia, from October 5 to 9, 2024. This article starts by briefly introducing the conference and sharing the findings from the Keynote Speech & Discussion that kicked off the conference. And we introduce some of the interesting research findings presented during the conference sessions.
We hope that through this article, many readers will have an indirect experience of the Ubicomp/ISWC conference and gain some good insights.
1. Introduction to Ubicomp/ISWC conferences
The Ubicomp/ISWC conference is a preeminent conference in the field of computer science and information systems, whose purpose is to explore how ubiquitous, pervasive, and wearable computing technologies integrate and impact people's daily lives.
A mainstay of the program at the conference is the presentation of papers published in Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), an academic journal that covers research on mobile, wearable, and ubiquitous technologies.
In addition to this, the conference offers Workshops and Tutorials, which provide an opportunity for researchers with the same research interests to discuss and learn together. Posters & Demos sessions provide a visual presentation of research.
2. Keynote Speech & Discussion: A Chance for Understanding the Research Direction Ubicomp Aims to Pursue
Attending the conference in person and listening to both the keynote and editors’ discussion was a great opportunity to directly experience and understand the goals and values of the community I belong to. The keynote speech provided a detailed explanation of how Ubicomp has evolved and the discussions that have shaped its current system. For example, it was fascinating to hear candid insights on why the reviewing process occurs every three months and the 'practical' issues behind maintaining the uniqueness of the conference name. The discussion of editors allowed me to listen directly to the perspectives of leading scholars on the direction of research that Ubicomp aims to pursue. Participating in this session helped me grasp the research focus of Ubicomp in the following presentation sessions. A key message that stood out was the recognition of potential harms in ubiquitous computing and the importance of addressing these issues through value-centered research. Overall, the keynote and discussion sessions were invaluable in deepening my understanding of the values and research directions that the Ubicomp community strives for.
3. Main Session: Paper Presentation
3.1. Health Monitoring Session
This session introduced studies that utilize a variety of mobile and sensor data to monitor specific diseases (e.g, hand edema treatment, dry eye, parkinson's, surgical outcomes, etc.) or health conditions (e.g., blood pressure, cybersickness, dental health, etc.). These sessions covered topics such as building systems that make health monitoring more accessible to users, developing models that leverage AI technologies to detect diseases and related symptoms early, and creating personalized wearable devices designed specifically for health monitoring. The related detailed sessions are as follows: Health Monitoring I (2B), Health Monitoring II (6B)
MediKnit: Soft Medical Making for Personalized and Clinician-Designed Wearable Devices for Hand Edema
Heather Jin Hee Kim, Narjes Pourjafarian, Arhan Choudhury, Joan Stilling, Hsin-Liu (Cindy) Kao
https://doi.org/10.1145/3678504
This paper presents an innovative approach to developing personalized wearable devices in the medical field, particularly suited for managing complex areas affected by edema, based on the concept of "soft medical making." The MediKnit system overcomes the limitations of traditional rigid, 3D-printed prototypes by employing a flexible, textile-based design tool to create customized devices tailored to the specific needs of hand edema. This differentiated approach considers the individual requirements and physical characteristics of each patient, offering better therapeutic effects for both healthcare providers and patients compared to conventional, one-size-fits-all medical devices. The main contributions can be summarized as follows: First, the clinician-centered design process allows for the creation of customized devices that reflect the insights of medical professionals. This approach goes beyond technical implementation, meeting the detailed requirements needed in real healthcare environments and allowing for more precise, patient-centered designs through collaboration with clinicians. Second, MediKnit provides an intuitive, easy-to-use design tool, enabling healthcare providers to design and modify devices without requiring technical expertise. This tool allows for quick and efficient fabrication, with the additional benefit of real-time adjustments based on patient feedback.
I feel this research is an exemplary case of advancing patient-centered treatment approaches through the integration of healthcare and HCI (Human-Computer Interaction). MediKnit has the potential to improve not only patient treatment experiences but also the work efficiency of healthcare providers. I look forward to seeing this technology evolve further, with the potential to expand beyond hand edema to manage a range of symptoms across different body areas.
Efficient and Robust Heart Rate Estimation Approach for Noisy Wearable PPG Sensors Using Ideal Representation Learning
Amashi Niwarthana, Pamuditha Somarathine, Pierre Qian, Ken-Tye Yong, Anusha Withana
https://doi.org/10.1145/3675095.3676606
This paper offers a significant contribution to the field of heart rate monitoring using photoplethysmography (PPG) by proposing a deep learning approach that effectively addresses noise issues commonly encountered in wearable devices. The main contributions can be summarized as follows: it introduces a new PPG signal denoising method by leveraging mathematically modeled ideal PPG waveforms in a GAN-based framework, which enhances heart rate estimation accuracy in noisy, real-world conditions. The model also demonstrates strong generalizability across various conditions; in a study with 22 participants performing diverse activities and using a new sensor, it consistently performs robustly across different noise sources and user conditions, outperforming state-of-the-art methods. Additionally, the model’s successful deployment on an Android device validates its practicality for mobile health applications, providing fast and reliable heart rate inferences even on resource-limited devices. Finally, with an efficient and lightweight model design, achieved by reducing the parameter count, it is suitable for wearable technology and adaptable for smaller devices like smartwatches.
I find this research to be an impressive showcase of how GANs and ideal signal modeling can transform real-time physiological monitoring in a practical and accessible way. Future work could further expand on these contributions by adapting this approach to estimate additional physiological metrics, such as blood oxygen levels or blood pressure, and by exploring compatibility with even more compact wearable devices.
UWB-enabled Sensing for Fast and Effortless Blood Pressure Monitoring
Zhi Wang, Beihong Jin, Fusang Zhang, Siheng Li, Junqi Ma
https://doi.org/10.1145/3659617
The paper presents a contactless blood pressure (BP) monitoring solution using Ultra-WideBand (UWB) technology, embodied in the RF-BP system, which leverages UWB signals to provide fast, non-invasive BP readings and addresses the discomfort and inconvenience of traditional cuff-based methods. By employing deep learning techniques to extract pulse wave features from UWB signals, the system accurately estimates systolic and diastolic BP, with strong results in both controlled and real-world settings. The main contributions include the development of RF-BP, a contactless BP monitoring system that uses UWB signals, enhancing user comfort and enabling frequent, effortless BP checks. The paper introduces advanced signal processing techniques—peak detection and principal component analysis (PCA)—to filter out non-pulse wave components from UWB signals, improving BP estimation accuracy by minimizing interference from user movement, respiration, and environmental noise. Additionally, the deep learning model is designed with a multi-scale feature extractor and attention-augmented residual blocks to capture BP-related features from UWB quasi-pulse waves, leading to precise and robust BP predictions. The RF-BP system is validated through comprehensive testing on both a public dataset and a self-built dataset with 70 participants, demonstrating strong performance metrics, including low mean absolute error (MAE) and standard deviation (SD) for systolic and diastolic BP predictions. Furthermore, the system’s performance in varied real-world environments and distances highlights RF-BP’s adaptability and reliability for practical use in daily settings, such as offices or homes.
Overall, This research is impressive because it effectively combines UWB signal processing and machine learning to tackle the challenges of traditional BP monitoring methods. The contactless nature of the RF-BP system enhances the user experience by making BP monitoring effortless and suitable for daily use, which could lead to higher user engagement and more consistent health tracking, especially for individuals with hypertension. I believe this paper demonstrates how advances in radar-based sensing and AI can meet real healthcare needs, providing valuable insights for further development in contactless health monitoring. Future research could focus on expanding this system to monitor additional vital signs or adapting it for even smaller and more portable devices, enhancing its applications and accessibility even further.
Predicting Multi-dimensional Surgical Outcomes with Multi-modal Mobile Sensing: A Case Study with Patients Undergoing Lumbar Spine Surgery
Ziqi Xu, Jingwen Zhang, Jacob Greenberg, Madelyn Frumkin, Saad Javeed, Justin K. Zhang, Braeden Benedict, Kathleen Botterbush, Thomas L Rodebaugh, Wilson Z. Ray, Chenyang Lu
https://doi.org/10.1145/3659628
This paper introduces �3TL, a multi-modal task learning model that predicts multiple outcomes after lumbar spine surgery using data from smartphones and wearable devices. The model addresses three key post-surgical outcomes: pain interference, physical function, and recovery quality. To achieve accurate predictions, the authors developed new features that quantify the association between self-reported measures (e.g., depression) and wearable data using the DSEM model. They further refined �3TL through a multi-task feature selection process, ensuring only relevant features were retained. As a result, �3TL not only showed good performance in single task classification but also in multiple task classification than other models.
I found this paper interesting because its approach could be directly applied to our lab's research on detecting mental health states using diverse sensor data. In our work, we rely on self-reported data as ground truth when predicting real-world mental health conditions such as stress, emotion, and depression. By adopting a similar methodology, we could harness the associations between past self-reported data and sensor data as predictors for current mental health states. Additionally, the model’s multi-task approach seems adaptable for simultaneously predicting various behavioral indicators related to mental health, like stress and bulimia.
PRECYSE: Predicting Cybersickness using Transformer for Multimodal Time-Series Sensor Data
Dayoung Jeong, Kyungsik Han
https://dl.acm.org/doi/abs/10.1145/3659594
The paper introduces transformer-based models designed to predict cybersickness in VR environments using multimodal time-series sensor data. The authors first collected data measuring cybersickness in VR by tracking eye movements, head movements, and physiological signals using an HMD and the E4 Wristband. They then developed a model that captures both temporal and spatial information in the sensor data. To explore effective input representations, they compared two embedding methods: one using normalized sensor data (0~1) as input (MS-STTN) and another converting the data into a spectrogram to encode spatiotemporal information as an image (MS-STTS). The results indicate that MS-STTS, which utilizes the spectrogram, outperformed existing models in predicting cybersickness across two VR cybersickness datasets.
I found this paper intriguing, particularly in its use of transformer-based models to capture the time-series characteristics of sensor data. The approach highlights the value of applying transformers to analyze temporal and spatial patterns in multimodal sensor data. However, I am concerned about the potential challenges this model might face in real-world applications, particularly regarding the considerable memory and processing power required for training transformers on extensive time-series data. Additionally, I observed that the choice of input data representation—whether normalized sensor data or a spectrogram—significantly impacts the model’s performance. This insight reinforced the importance of thoughtfully organizing input data when building affect detection models, as it can have a substantial effect on prediction accuracy.
Bio
Author 1. Hansoo Lee
Hansoo Lee is a Ph.D. student in the School of Computing at KAIST. His current research focuses on human-computer interaction, specifically on quantifying human routine behaviors through mining modeling and monitoring mental health states with an explainable digital phenotyping approach. Through this research, he explores ways to enhance the explainability and performance of mental health models by leveraging smartphone-based mining of human routine behaviors, aiming to reduce data types and volume for privacy preservation and faster model building.
Author 2. Heejeong Lim
Heejeong Lim is a Master's student in the Graduate School of Data Science at KAIST. Her research focuses on affective computing, specifically on building an affect state detection model using multimodal data. Currently, she is exploring building a model to measure emotion suppression for emotion workers using real-world multimodal data.
3.2. Human Activity Recognition (HAR)
These sessions covered topics such as collecting new datasets to recognize human actions through various sensors, developing related models, and introducing applications that utilize HAR. The related detailed sessions are as follows: Human Activity Recognition I (5A) and Human Activity Recognition II (9A)
SmartDampener: An Open Source Platform for Sport Analytics in Tennis
Runze Liu, Taiting Lu, Shengming Yuan, Hao Zhou, Mahanth Gowda
https://doi.org/10.1145/3678507
This study involved a custom sensor attached to the dampener of a tennis racket, allowing automatic measurement of tennis movement metrics to analyze athlete performance. Initially, I was concerned that the dampener might be susceptible to damage from tennis balls, which could affect sensor durability. However, the researchers demonstrated the sensor's versatility by mounting it on different parts of the racket, such as the head or frame. Additionally, they proposed potential applications for other racket sports, which highlights the broader relevance of their work. I would recommend this study to individuals interested in quantitative measurement of physical performance through behavioral sensors or those with a personal interest in various racket sports (like myself!).
IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition
Zikang Leng, Amitrajit Bhattacharjee, Hrudhai Rajasekhar, Lizhe Zhang, Elizabeth Bruda, Hyeokhyen Kwon, Thomas Ploetz
https://doi.org/10.1145/3678545
This study proposed a novel and intriguing approach to constructing a dataset for Human Activity Recognition (HAR). The researchers utilized ChatGPT 2.0 to generate descriptive expressions for various actions across different contexts and automatically matched these descriptions with motion data to create HAR datasets. Two main challenges were identified: first, determining the saturation point in generating motion descriptions; and second, addressing hallucinations in generated data that did not align accurately with the descriptions. The authors tackled these challenges with a custom algorithm to establish saturation points and applied motion filters for data alignment. This research is timely and compelling, as it explores how generative language models like ChatGPT can be applied from the sensor data community's perspective to address issues in HAR research. I believe this study offers significant potential for generating diverse datasets, especially when real-world data collection proves challenging. However, it is essential to remember that the algorithm and methodology may require optimization based on specific contexts, presenting new challenges and research opportunities.
Kirigami: Lightweight Speech Filtering for Privacy-preserving Activity Recognition using Audio
Sudershan Boovaraghavan, Haozhe Zhou, Mayank Goel, Yuvraj Agarwal
https://doi.org/10.1145/3643502
While audio data offers clear benefits, this study addressed privacy concerns by developing an on-device speech filtering system. To overcome vulnerabilities in previous filtering methods, the researchers designed a filter that protects privacy by preventing reconstruction, while still allowing essential HAR information to be extracted. Notably, the system is lightweight and operates offline, making it possibly well-suited for on-device models. The code is available on GitHub (https://github.com/synergylabs/kirigami), providing a valuable resource for those interested in ethical audio data use.
Bio
Author. Duri Lee
She is a Ph.D. student in the School of Computing at KAIST. Her current research focuses on human-computer interaction, specifically on quantifying user behaviors (human activity recognition) and assessing affective states that influence decision-making. Through her research on self-awareness using multimodal data, she is exploring ways to develop applications that mediate emotional responses to improve the quality of decisions.
3.3. Human Tracking and Monitoring
The session focuses on research related to human tracking and monitoring, which includes human mental state, human shape, and human health prediction. To be specific, this session covers stress, mood, social context, energy expenditure, sleep stage, Parkinson’s Disease, 3D human posture and shape, and bed occupancy prediction. The related detailed session is as follows: Human Tracking and Monitoring (8B)
M3BAT: Unsupervised Domain Adaptation for Multimodal Mobile Sensing with Multi-Branch Adversarial Training
Lakmal Meegahapola, Hamza Hassoune, Daniel Gatica-Perez
https://doi.org/10.1145/365959
The paper addresses the challenge of distribution shifts in multimodal mobile sensing data when deploying models in real-world scenarios. The authors propose a novel approach called M3BAT (Multi-Branch Adversarial Training) to improve domain adaptation for multimodal data by leveraging multiple branches in their architecture. This method extends domain adversarial neural networks (DANN) to better handle multimodal sensor data through the introduction of multiple feature-specific branches. To support this approach, the authors conducted a statistical analysis on multimodal datasets to quantify distribution shifts across different data modalities, using two datasets (WENET and WEEE) to identify which modalities exhibit significant differences. M3BAT’s multi-branch architecture is designed with branches tailored to specific feature sets based on their distribution shifts, enhancing the unsupervised domain adaptation (UDA) process by enabling stable training that adapts to different levels of shifts—high, moderate, and low. The effectiveness of this approach was demonstrated on two multimodal mobile sensing datasets across three inference tasks and 14 source-target domain pairs, covering both classification and regression. Results showed substantial improvements, including up to a 12% increase in AUC for classification and a 0.13 reduction in MAE for regression tasks, compared to models trained solely on source data.
Overall, this paper did a systematic study on multiple datasets proposing some new algorithm for unsupervised domain adaptation which is quite valuable to the community. I think this paper is an important milestone towards domain adaptation after the [GLOBEM cross-dataset generalization paper](https://doi.org/10.1145/3569485) which systematically evaluates domain generalization techniques in the mobile sensing field.
However, further efforts are needed to analyze supervised domain adaptation techniques in addition to unsupervised methods. As highlighted in the paper, the current work represents an initial exploration of domain adaptation, focusing on just one specific type of approach. To advance the field, comprehensive research should be undertaken to systematically evaluate a broader range of domain adaptation techniques for multimodal mobile sensing.
Self-Supervised Representation Learning and Temporal-Spectral Feature Fusion for Bed Occupancy Detection
Yingjian Song, Zaid Farooq Pitafi, Fei Dou, Jin Sun, Xiang Zhang, Bradley G Phillips, WenZhan Song
https://doi.org/10.1145/3678514
This paper presents a novel method called SeismoDot for detecting bed occupancy in sleep monitoring systems. Bed occupancy detection is crucial for sleep monitoring systems to ensure accuracy in subsequent tasks such as heart rate and respiration monitoring. Existing methods often lack generalizability due to reliance on threshold-based approaches and require extensive labeled data, making practical implementation challenging. To address these issues, the SeismoDot framework introduces a self-supervised learning module that learns robust representations without requiring pre-training on large labeled datasets; unlike conventional methods, this module is co-optimized with the primary detection task, ensuring that learned features remain relevant. Additionally, a spectral-temporal feature fusion module is incorporated to integrate temporal and spectral features, enhancing the model’s generalizability by effectively leveraging diverse data from both domains. The main contributions of SeismoDot include the use of seismic sensors, offering a privacy-friendly solution compared to camera-based approaches, the integration of self-supervised learning directly into the main task to avoid the need for separate pre-training, and the effective combination of temporal and spectral data through feature fusion, addressing limitations associated with relying solely on one type of data. Overall, this paper also tries to handle the generalizability issue though through self-supervised learning methods. And the system archives 98.08% F1 score across 13 diverse environments which proves the efficacy of the proposed system.
Recently, generalizability has become a major bottleneck in the field of mobile and ubiquitous sensing due to the heterogeneous nature of human behavior and devices. Greater efforts are needed to explore domain adaptation and generalization techniques. Moreover, labeled data, which is crucial for model development, is often challenging to collect. Therefore, semi-supervised and self-supervised learning approaches are also vital for advancing this field.
Bio
Autor. Panyu Zhang
Panyu Zhang is currently a doctoral student at KAIST Graduate School of Data Science. His research mainly focuses on mobile affective sensing. He is interested in the reproducibility and generalizability of mobile affective sensing. Through exploring reproducibility and generalizability, he is devoted to creating a common baseline in the field that can be generalized across different users.
3.4. Fitness and Sports
At this year’s Ubicomp/ISWC conference, there was significant interest in studies that monitor users’ conditions during fitness and sports activities using various sensor data. In particular, "methods for real-time analysis of users' breathing during exercise" received much attention. These technologies aim to improve the accuracy of health monitoring during exercise and help prevent potential risks. The related detailed session is as follows: Fitness and Sports (10B)
DeepBreath: Breathing Exercise Assessment with a Depth Camera
Wentao Xie, Chi Xu, Yanbin Gong, Yu Wang, Yuxin Liu, Jin Zhang, Qian Zhang, Zeguang Zheng, Shifang Yang
https://doi.org/10.1145/3678519
DeepBreath presents a breathing exercise assessment system using a depth camera. The system was developed to help patients with chronic obstructive pulmonary disease (COPD) perform effective breathing exercises at home. The authors identified limitations in traditional methods of measuring breathing modes (chest or belly breathing) and lung volume, and aimed to address these challenges. DeepBreath measures lung volume using a depth camera and applies AI to analyze breathing patterns and lung volume simultaneously. In a study with 22 healthy adults and 14 COPD patients, the system achieved a 92% accuracy in classifying breathing modes. This system simplifies the process of monitoring breathing exercises at home and offers greater accuracy compared to previous methods.
JoulesEye: Energy Expenditure Estimation and Respiration Sensing from Thermal Imagery While Exercising
Rishiraj Adhikary, Maite Sadeh, Nipun Batra, Mayank Goel
https://doi.org/10.1145/3631422
JoulesEye explores a system that uses thermal imaging to estimate calorie expenditure and monitor breathing during exercise. The research team focused on the inaccuracies of heart rate-based calorie estimation methods used in smartwatches, and developed a more precise system that relies on respiratory rates. JoulesEye employs a low-resolution thermal camera attached to a smartphone to detect temperature changes around the nose caused by breathing. These changes are used to estimate both the respiratory rate and calorie expenditure. In an experiment involving 54 participants performing cycling and running exercises, JoulesEye outperformed commercial smartwatches by over 30% in accuracy. The system demonstrates high potential for integration into future wearable devices.
RF-GymCare: Introducing Respiratory Prior for RF Sensing in Gym Environments
Jianyang Wang, Dongheng Zhang, Binbin Zhang, Jinbo Chen, Yang Hu, Yan Chen
https://doi.org/10.1145/3678568
RF-GymCare focuses on using radio frequency (RF) sensors to monitor breathing in gym environments. The study addresses the challenge of separating physical movement signals from respiratory signals during high-intensity exercises and proposes a blind source separation (BSS) method that prioritizes respiratory signals. RF-GymCare leverages respiratory-related knowledge to separate mixed signals during exercise. Tested across nine types of exercises with 13 hours of collected data, the system outperformed existing RF monitoring methods in accurately isolating respiratory signals. This advancement makes it possible to monitor breathing even in complex gym environments.
All three papers focus on improving the accuracy and feasibility of breathing monitoring approaches by employing depth cameras, thermal imaging, and RF sensors to measure respiratory rates accurately and apply the results in various contexts. What these studies share is a commitment to making these technologies more applicable in real-life settings, overcoming the limitations of traditional methods by employing various mobile sensor technologies. In the future, these advancements are expected to be used across multiple exercise contexts, enabling better health management and safer workouts through real-time breathing monitoring.
Bio
Author. Gyuwon Jung
Gyuwon Jung is a Ph.D. candidate in the School of Computing at KAIST, specializing in Human-Computer Interaction (HCI). His research focuses on empowering users to understand themselves through everyday data collected from mobile devices, with a particular interest in exploring relationships between behaviors, contexts, and health states. His work aims to infer causalities and counterfactual scenarios from this data and deliver meaningful insights through personal informatics systems. His research spans mobile data, causal inference, counterfactual explanations, digital therapeutics, behavior change, and user engagement.
4. Learning
We would like to conclude this article with brief remarks from the authors who contributed to this article.
Hansoo
Participating in UbiComp 2024 was an enriching experience, allowing me to engage with the latest advancements in ubiquitous computing and connect with researchers across diverse fields. The conference provided valuable insights into emerging trends and innovative applications that align closely with my own work in human-computer interaction and digital phenotyping.
Heejeong
Attending UbiComp was valuable to me, as discussing with researchers in related fields offered meaningful insights. It was also a great opportunity to understand research trends and explore how approaches from other fields might be applied to my research.
Duri
Attending the conference provided me with an invaluable opportunity to gain insights into the key ideas currently being actively discussed in our field, while also reflecting on my identity as part of this research community.
It was also a great chance to connect with researchers working on fascinating topics, and I found it particularly inspiring to see others addressing the same questions I had been curious about.
Panyu
The UbiComp event is quite exciting, and meeting researchers in related domains for insightful discussions is valuable, as it can prevent us from reinventing the wheel. After attending the Ubicomp sessions, I realized that small datasets and lack of generalizability are common challenges across the entire field.
Gyuwon
Attending Ubicomp was a great opportunity to share my research, connect with experts in the field, and receive valuable feedback that helped me think about the future directions of my research. I realized that while enhancing sensing techniques and improving model performance are important, it is also equally essential to consider the practical benefits these advancements bring to users deeply.
