AR in Surgery - iOS & Apple Vision Pro|
This project connects to "AR in Surgery - Hololens 2".
Beyond the HoloLens 2, iOS devices and the newly released Apple Vision Pro offer promising navigation capabilities during surgeries because of their diverse AR features.
The Apple Vision Pro is particularly intriguing due to its sharp and precise display of AR content, while iOS systems are relevant primarily because of their widespread availability. A precise and intuitive navigation system on these platforms could be a groundbreaking advancement in medicine.
Technologies
- visionOS (Swift)
- iOS (Swift)
- RealityKit
- ARKit
Challenges
From the features perspective, the biggest challenges were the same as on the Hololens 2, like an accurate relationship between virtual content and real-world objects. But because the necessary calculations had already been implemented on the Hololens, the same calculations could now be used and ported to iOS and visionOS.
Developing on two operate systems that are similar, but not the same needed good organization and a well-structured codebase. Because the Apple Vision was still unreleased in Germany, the development and testing had to be mostly improvised on the available simulator. Especially the required extensive communication between SwiftUI and RealityKit turned out to be challenging to implement.
Results
The outcome of the development for iOS is an intuitive and compact prototype to bring surgical navigation abilities to common devices for the first time and therefore offer incredible new use cases.
A working prototype on visionOS leads to promising opportunities in the near future. With the Apple Vision, there are various possibilities to increase the system's accuracy and functionality because of its state-of-the-art technology.
Both operating systems now support the same functionalities as their counterpart on the Hololens.
Screenshots
