Full-coverage Robotic Liver Ultrasound Scan with RL-based Path Planning (Ongoing)

Ultrasound waves are reflected by the high-density bone tissue, causing a "shadow" effect that obscures the underlying structures. Therefore, the robotic ultrasound scan of the liver area presents a significant challenge due to the ribs obscuring parts of the liver, making it difficult to fully scan for tumors. In addressing this problem, this ongoing project in TUM's CAMP chair (Chair for Computer Aided Medical Procedures & Augmented Reality) tries to employ deep reinforcement learning algorithms, seeking to optimize the robotic arm's movements to navigate around the ribcage and ensure complete and efficient coverage of the liver tumor area during ultrasound scans. The demos are displayed below:

Demo videos: Tumor (red), ultrasound ray (gray) and ribs (yellow)

Dual-arm Manipulation (Ongoing)

The "Adaptive Process Planning Using Mobile Pre-Assembly Systems" project, sponsored by Kompetenzzentrum Mittelstand GmbH (https://kme-mittelstand.de/), is focused on the creation of a dual-arm robotic system specifically designed for intralogistics applications. This initiative seeks to broaden the capabilities of automated guided vehicles (AGVs) by incorporating complex handling and pre-assembly processes. By performing these tasks while the mobile platform is in motion, the project aims to enhance the efficiency of real-world logistics applications, representing a significant advancement in the field of automated material handling.

I am now working as a working student for this project. My task mainly includes the development of process control for the robot, the state and action interface in ROS.

Improving Patient Safety through Movement Tracking ​during Telerobotic Examinations​

The uneven development of medical services in different regions, coupled with the need to protect doctors from the risk of contracting infectious diseases, has led to the gradual growth of remote robotic medical diagnosis. However, when utilizing robots for remote diagnosis, patients, especially elderly, children or pregnant women, may perform dangerous actions due to their confusion, curiosity, or anxiety. So, in this project with MITI (Minimally invasive Interdisciplinary Therapeutical Interventions) group, we aim to address the issue of potential injury specifically caused by sudden movements.

Here is our pipeline. Firstly, we have a camera on the top that captures both RGB and depth scene images. Then, the image is processed by a human pose detection model to identify key points of the patient. By analyzing the consecutive frames, the system evaluates the motion of each body joint and detects any risky motion patterns that might indicate a potential collision. If there is any detected risky motion, the system immediately retreats the robot arm following a safe trajectory. Otherwise, the examination continues. The demos are displayed below:

Demo video: Notice that, the human pose detection model is based on OpenPose library (developed by CMU)