Building a Healthcare Robot from Simulation to Deployment with NVIDIA Isaac

TL;DR

A hands-on guide to collecting data, training policies, and deploying autonomous medical robotics workflows on real hardware

Table-of-Contents

• Building a Healthcare Robot from Simulation to Deployment with NVIDIA Isaac
  • TL;DR
  • Table-of-Contents
  • Introduction
  • SO-ARM Starter Workflow; Building an Embodied Surgical Assistant
    • Technical Implementation
    • Sim2Real Mixed Training Approach
    • Hardware Requirements
    • Data Collection Implementation
    • Simulation Teleoperation Controls
    • Model Training Pipeline
  • End-to-End Sim Collect–Train–Eval Pipelines
    • Generate Synthetic Data in Simulation
    • Train and Evaluate Policies
    • Convert Models to TensorRT
  • Getting Started
    • Resources

Introduction

Simulation has been a cornerstone in medical imaging to address the data gap. However, in healthcare robotics until now, it's often been too slow, siloed, or difficult to translate into real-world systems.

NVIDIA Isaac for Healthcare, a developer framework for AI healthcare robotics, enables healthcare robotics developers in solving these challenges via offering integrated data collection, training, and evaluation pipelines that work across both simulation and hardware. Specifically, the Isaac for Healthcare v0.4 release provides healthcare developers with an end-to-end SO - ARM based starter workflow and the bring your own operating room tutorial. The SO-ARM starter workflow lowers the barrier for MedTech developers to experience the full workflow from simulation to train to deployment and start building and validating autonomous on real hardware right away.

In this post, we'll walk through the starter workflow and its technical implementation details to help you build a surgical assistant robot in less time than ever imaginable before.

SO-ARM Starter Workflow; Building an Embodied Surgical Assistant

The SO-ARM starter workflow introduces a new way to explore surgical assistance tasks, and providing developers with a complete end-to-end pipeline for autonomous surgical assistance: