Skip to content

DeveloperGuide.md

Latest commit

 

History

History
259 lines (184 loc) · 14.8 KB

DeveloperGuide.md

File metadata and controls

259 lines (184 loc) · 14.8 KB

Information for developers

This Slicer extension is in active development. The API may change from version to version without notice.

Build instructions

  • Build the extension against the newly built Slicer using the SuperBuild system.
  • To start Slicer from a build tree and ensure the extension is properly loaded, consider running the SlicerWithVirtualReality launcher. For more details, see here.

CMake build options

The top-level CMakeLists.txt exposes three backend options:

CMake option Default (Windows) Default (macOS) Description
SlicerVirtualReality_HAS_OPENVR_SUPPORT ON OFF Build the OpenVR XR backend
SlicerVirtualReality_HAS_OPENXR_SUPPORT ON OFF Build the OpenXR XR backend
SlicerVirtualReality_HAS_OPENXRREMOTING_SUPPORT ON OFF Build OpenXR Remoting support (HoloLens 2)

OpenXR Remoting is automatically disabled if SlicerVirtualReality_HAS_OPENXR_SUPPORT is OFF. It is only supported on Windows.

Key classes

Class Location Description
vtkMRMLVirtualRealityViewNode VirtualReality/MRML/ MRML node holding all VR view settings (backend, magnification, controller transforms, etc.)
vtkSlicerVirtualRealityLogic VirtualReality/Logic/ Main logic class: activates/deactivates VR, manages the active view node, and sets up button bindings
qMRMLVirtualRealityView VirtualReality/Widgets/ Qt widget that owns the VTK render window and interactor for the VR view
vtkVirtualRealityViewInteractorObserver VirtualReality/MRMLDM/ Bridges VTK VR interactor events to Slicer displayable managers
vtkVirtualRealityViewInteractorStyleDelegate VirtualReality/MRMLDM/ Shared delegate implementing scene/object grab and gesture logic for both OpenVR and OpenXR styles
vtkVirtualRealityComplexGestureRecognizer VirtualReality/MRMLDM/ Slicer-specific two-controller gesture recognition (translate/rotate/scale)
vtkMRMLVirtualRealityViewDisplayableManagerFactory VirtualReality/MRMLDM/ Singleton factory that registers displayable managers for the VR view

Mapping of Controller Action to VTK event

The mapping process consists of several: action manifest json file maps controller-specific interaction path to VTK event path, then VTK render window interactor maps VTK event path to VTK event, which is processed by interactor style, which may be further customized by style delegates. For low-level custom processing of events, it is possible to intercept VTK events in the interactor.

1. Mapping from interaction path to VTK event path

Mapping from interaction path to VTK event path is specified in the action manifest file. Parsing the vtk_open<vr|xr>_actions.json action manifest file to link controller-specific interaction paths (e.g., /user/hand/right/input/b) with generic event paths (e.g., showmenu). This file references controller-specific binding files, usually named vtk_open<vr|xr>_binding_<vendor_name>.json, where each controller interaction path is associated with a VTK event path.

Action Manifest File

The controller interaction paths are specific to each backend:

As of Slicer@c7fe8657c, the provided vtk_open<vr|xr>_actions.json and vtk_open<vr|xr>_binding_<vendor_name>.json files in the vtkRenderingOpenVR and vtkRenderingOpenXR VTK modules are as follow:

OpenVR OpenXR
Action manifest url url
- HP Motion Controller url url
- HTC Vive Controller url url
- Microsoft Hand Interaction url
- Oculus Touch (Meta Quest) url url
- Valve Knuckles url url
- Khronos Simple Controller1 url

These files serve as essential references for mapping controller actions to VTK events.

2. Mapping from VTK event path to VTK event

Assigning a VTK event path (e.g., showmenu) to either a VTK event (e.g., vtk.vtkCommand.Menu3DEvent) or a std::function for a single controller is carried out in vtkOpen<VR|XR>InteractorStyle::SetupActions().

Action Identifier Differences Between Backends

OpenVR and OpenXR use different formats for action identifiers when calling vtkVRRenderWindowInteractor::AddAction():

Backend Format Example
OpenVR Full action path /actions/vtk/in/TriggerAction
OpenXR Action name only (lowercase) triggeraction

The vtkSlicerVirtualRealityLogic::SetGestureButton*() helpers detect the active backend at runtime using vtkOpenXRRenderWindowInteractor class name and apply the correct identifier automatically. Additionally, OpenXR grip/squeeze button bindings are inconsistent across controllers (the right grip is typically bound to positionprop, while the left grip may be bound to complexgestureaction or not bound at all), so both identifiers are registered when using OpenXR.

The default button configuration (set in qMRMLVirtualRealityViewPrivate::createRenderWindow()) is:

  • Trigger button: grab objects and world
  • Grip button: complex gesture (translate/rotate/scale scene)

Complex Gesture Support

Recognition of complex gesture events commences when the two controller buttons mapped to the ComplexGesture action are pressed.

The SlicerVirtualReality implements its own heuristic by specializing the HandleComplexGestureEvents() and RecognizeComplexGesture() in the vtkVirtualRealityComplexGestureRecognizer class.

Limitations:

  • The selected controller buttons are exclusively mapped to the ComplexGesture action and cannot be associated with a regular action.

  • To workaround an OpenVR specific limitation, each button expected to be involved in the complex gesture needs to be respectively associated with /actions/vtk/in/ComplexGestureAction and /actions/vtk/in/ComplexGestureAction_Event2.

Low-level interception of events

For implementing completely custom behavior, mapping from VTK event path to VTK event can be customized (by using slicer.modules.virtualreality.logic().AddAction()) and the VTK event can be intercepted in the render window interactor by adding a high-priority observer.

Useful Python Snippets

Activate virtual reality view

import logging
import slicer

def isXRBackendInitialized():
    """Determine if XR backend has been initialized."""
    vrLogic = slicer.modules.virtualreality.logic()
    return vrLogic.GetVirtualRealityActive() if vrLogic else False

def vrCamera():
    # Get VR module widget
    if not isXRBackendInitialized():
        return None
    # Get VR camera
    vrViewWidget = slicer.modules.virtualreality.viewWidget()
    if vrViewWidget is None:
      return None
    rendererCollection = vrViewWidget.renderWindow().GetRenderers()
    if rendererCollection.GetNumberOfItems() < 1:
        logging.error('Unable to access VR renderers')
        return None
    return rendererCollection.GetItemAsObject(0).GetActiveCamera()


assert isXRBackendInitialized() is False
assert vrCamera() is None

vrLogic = slicer.modules.virtualreality.logic()
vrLogic.SetVirtualRealityActive(True)

assert isXRBackendInitialized() is True
assert vrCamera() is not None

Set virtual reality view background color to black:

color = [0,0,0]
vrView=getNode('VirtualRealityView')
vrView.SetBackgroundColor(color)
vrView.SetBackgroundColor2(color)

Set whether a node can be selected/grabbed/moved:

nodeLocked.SetSelectable(0)
nodeMovable.SetSelectable(1)

Low-level event handling

# Get the render window interactor
vrViewWidget = slicer.modules.virtualreality.viewWidget()
interactor = vrViewWidget.interactor()

# Set the gesture button to none to allow custom event mapping
# (otherwise the squeeze button would be unavailable for custom mapping by default)
slicer.modules.virtualreality.logic().SetGestureButtonToNone(interactor)

# Use high priority observers to ensure we get to process the event before the interactor style (and we can prevent
# any further processing of the event)
highPriority = 100.0

# By default, right-hand trigger button is mapped to `triggeraction`, which then calls `Select3DEvent`.
# Here we take over the trigger button:

@vtk.calldata_type(vtk.VTK_OBJECT)
def onSelect3DEvent(caller, event, calldata):
    print(f"Select3DEvent received: {event}")
    print(f"WorldPosition: {calldata.GetWorldPosition()}")    
    # Prevent further processing
    caller.GetCommand(select3DObserverTag).AbortFlagOn()

select3DObserverTag = interactor.AddObserver("Select3DEvent", onSelect3DEvent, highPriority)

# Take over the right-hand joystick:

@vtk.calldata_type(vtk.VTK_OBJECT)
def onViewerMovement3DEvent(caller, event, calldata):
    print(f"ViewerMovement3DEvent received: {event}")
    print(f"TrackPadPosition: {calldata.GetTrackPadPosition()}")

interactor.AddObserver("ViewerMovement3DEvent", onViewerMovement3DEvent, highPriority)

# In recent software versions, it is also possible to modify the VTK event path to VTK event mapping.
# For example: we can map the trigger button to the menu event.

slicer.modules.virtualreality.logic().AddAction(interactor, "triggeraction", vtk.vtkCommand.Menu3DEvent, False)

The default event mapping for Meta Quest (Oculus Touch) is incomplete and not very intuitive. It is often useful to make these changes/additions (map A button to a rarely used VTK event that can be customized; map trigger button to triggeraction for more intuitive naming, add mapping of grab button, by default to move a node):

        {
          "inputs": {
            "click": {
              "output": "nextcamerapose"
            }
          },
          "path": "/user/hand/right/input/a"
        },
        {
          "inputs": {
            "value": {
              "output": "triggeraction"
            }
          },
          "path": "/user/hand/right/input/trigger"
        },
        {
          "inputs": {
            "value": {
              "output": "positionprop"
            }
          },
          "path": "/user/hand/right/input/squeeze"
        },

Related VTK modules

Footnotes

  1. https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#_khronos_simple_controller_profile ↩