diff --git a/CHANGES.md b/CHANGES.md index d8030b31..83e48b57 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -18,6 +18,7 @@ - Add feature flags to Khronos loader for HTC - Add XR_HTC_passthrough extension wrapper - Add manifest entries to Pico and switch Pico to using the Khronos Loader +- Add Meta Passthrough tutorial doc ## 2.0.3 - Migrate the export scripts from gdscript to C++ via gdextension diff --git a/docs/manual/meta/img/passthrough/bcs_passthrough_filter.jpg b/docs/manual/meta/img/passthrough/bcs_passthrough_filter.jpg new file mode 100644 index 00000000..1f762810 Binary files /dev/null and b/docs/manual/meta/img/passthrough/bcs_passthrough_filter.jpg differ diff --git a/docs/manual/meta/img/passthrough/color_lut_passthrough_filter.jpg b/docs/manual/meta/img/passthrough/color_lut_passthrough_filter.jpg new file mode 100644 index 00000000..4a20f132 Binary files /dev/null and b/docs/manual/meta/img/passthrough/color_lut_passthrough_filter.jpg differ diff --git a/docs/manual/meta/img/passthrough/color_map_passthrough_filter.jpg b/docs/manual/meta/img/passthrough/color_map_passthrough_filter.jpg new file mode 100644 index 00000000..fd6af53d Binary files /dev/null and b/docs/manual/meta/img/passthrough/color_map_passthrough_filter.jpg differ diff --git a/docs/manual/meta/img/passthrough/edge_color_passthrough_filter.jpg b/docs/manual/meta/img/passthrough/edge_color_passthrough_filter.jpg new file mode 100644 index 00000000..10619f7d Binary files /dev/null and b/docs/manual/meta/img/passthrough/edge_color_passthrough_filter.jpg differ diff --git a/docs/manual/meta/img/passthrough/mono_map_passthrough_filter.jpg b/docs/manual/meta/img/passthrough/mono_map_passthrough_filter.jpg new file mode 100644 index 00000000..829ea1c3 Binary files /dev/null and b/docs/manual/meta/img/passthrough/mono_map_passthrough_filter.jpg differ diff --git a/docs/manual/meta/img/passthrough/node_tree_passthrough_geometry.png b/docs/manual/meta/img/passthrough/node_tree_passthrough_geometry.png new file mode 100644 index 00000000..8197c0cd Binary files /dev/null and b/docs/manual/meta/img/passthrough/node_tree_passthrough_geometry.png differ diff --git a/docs/manual/meta/img/passthrough/passthrough_color_lut_resource.png b/docs/manual/meta/img/passthrough/passthrough_color_lut_resource.png new file mode 100644 index 00000000..b0075d65 Binary files /dev/null and b/docs/manual/meta/img/passthrough/passthrough_color_lut_resource.png differ diff --git a/docs/manual/meta/img/passthrough/passthrough_color_map_resource.png b/docs/manual/meta/img/passthrough/passthrough_color_map_resource.png new file mode 100644 index 00000000..5bf117cd Binary files /dev/null and b/docs/manual/meta/img/passthrough/passthrough_color_map_resource.png differ diff --git a/docs/manual/meta/img/passthrough/passthrough_export_settings.png b/docs/manual/meta/img/passthrough/passthrough_export_settings.png new file mode 100644 index 00000000..f0b4c907 Binary files /dev/null and b/docs/manual/meta/img/passthrough/passthrough_export_settings.png differ diff --git a/docs/manual/meta/img/passthrough/passthrough_mono_map_resource.png b/docs/manual/meta/img/passthrough/passthrough_mono_map_resource.png new file mode 100644 index 00000000..610665e7 Binary files /dev/null and b/docs/manual/meta/img/passthrough/passthrough_mono_map_resource.png differ diff --git a/docs/manual/meta/img/passthrough/projected_passthrough.jpg b/docs/manual/meta/img/passthrough/projected_passthrough.jpg new file mode 100644 index 00000000..758705bd Binary files /dev/null and b/docs/manual/meta/img/passthrough/projected_passthrough.jpg differ diff --git a/docs/manual/meta/img/passthrough/projected_passthrough_hole_punch.jpg b/docs/manual/meta/img/passthrough/projected_passthrough_hole_punch.jpg new file mode 100644 index 00000000..cab2b4ab Binary files /dev/null and b/docs/manual/meta/img/passthrough/projected_passthrough_hole_punch.jpg differ diff --git a/docs/manual/meta/img/passthrough/reconstruction_passthrough.jpg b/docs/manual/meta/img/passthrough/reconstruction_passthrough.jpg new file mode 100644 index 00000000..72e63b50 Binary files /dev/null and b/docs/manual/meta/img/passthrough/reconstruction_passthrough.jpg differ diff --git a/docs/manual/meta/img/export_settings.png b/docs/manual/meta/img/scene_manager/export_settings.png similarity index 100% rename from docs/manual/meta/img/export_settings.png rename to docs/manual/meta/img/scene_manager/export_settings.png diff --git a/docs/manual/meta/img/global_mesh.png b/docs/manual/meta/img/scene_manager/global_mesh.png similarity index 100% rename from docs/manual/meta/img/global_mesh.png rename to docs/manual/meta/img/scene_manager/global_mesh.png diff --git a/docs/manual/meta/img/global_mesh_wireframe.png b/docs/manual/meta/img/scene_manager/global_mesh_wireframe.png similarity index 100% rename from docs/manual/meta/img/global_mesh_wireframe.png rename to docs/manual/meta/img/scene_manager/global_mesh_wireframe.png diff --git a/docs/manual/meta/img/scene_manager_options.png b/docs/manual/meta/img/scene_manager/scene_manager_options.png similarity index 100% rename from docs/manual/meta/img/scene_manager_options.png rename to docs/manual/meta/img/scene_manager/scene_manager_options.png diff --git a/docs/manual/meta/img/scene_tree.png b/docs/manual/meta/img/scene_manager/scene_tree.png similarity index 100% rename from docs/manual/meta/img/scene_tree.png rename to docs/manual/meta/img/scene_manager/scene_tree.png diff --git a/docs/manual/meta/img/spatial_entities.png b/docs/manual/meta/img/scene_manager/spatial_entities.png similarity index 100% rename from docs/manual/meta/img/spatial_entities.png rename to docs/manual/meta/img/scene_manager/spatial_entities.png diff --git a/docs/manual/meta/img/spatial_entity_scene.png b/docs/manual/meta/img/scene_manager/spatial_entity_scene.png similarity index 100% rename from docs/manual/meta/img/spatial_entity_scene.png rename to docs/manual/meta/img/scene_manager/spatial_entity_scene.png diff --git a/docs/manual/meta/index.rst b/docs/manual/meta/index.rst index 7714e7c5..810aef36 100644 --- a/docs/manual/meta/index.rst +++ b/docs/manual/meta/index.rst @@ -8,4 +8,5 @@ Meta :name: toc-meta scene_manager + passthrough diff --git a/docs/manual/meta/passthrough.rst b/docs/manual/meta/passthrough.rst new file mode 100644 index 00000000..a9c84768 --- /dev/null +++ b/docs/manual/meta/passthrough.rst @@ -0,0 +1,248 @@ +Meta Passthrough +================ + +If your project requires the user to see their physical environment, you'll need to use the headset's passthrough functionality. +The Godot OpenXR Vendors Plugin provides access to several passthrough modes and filters supported on the Meta Quest, all of which will be broken down in this tutorial. + +Export Settings +----------------- + +In order to utilize passthrough, you'll first have to enable the feature in your project's Export Settings. Open up **Export Settings**, navigate to **Meta XR Features**, +and locate the **Passthrough** setting. Changing this to either **Optional** or **Required** will allow a user's headset to utilize passthrough if permission is granted. + +.. image:: img/passthrough/passthrough_export_settings.png + +.. _reconstruction-passthrough: +Reconstruction Passthrough +-------------------------- + +.. figure:: img/passthrough/reconstruction_passthrough.jpg + :align: center + + Example of unfiltered reconstruction passthrough imagery (Quest Pro) + +Reconstruction passthrough will fill the entire background with imagery of the user's surrounding environment, thus "reconstructing" the space outside of the headset. +To enable this passthrough mode, ensure the following: + +* The viewport's ``transparent_bg`` property is set to ``true``. +* The environment's ``background_mode`` property is set to ``Environment.BG_COLOR``. +* The environment's ``background_color`` property should have an alpha value of ``0.0``. +* The `OpenXRInterface `_'s ``environment_blend_mode`` property is set to ``XRInterface.XR_ENV_BLEND_MODE_ALPHA_BLEND``. + +The ``OpenXRInterface`` parent class `XRInterface `_ has a method ``get_supported_environment_blend_modes()`` +that will list ``XRInterface.XR_ENV_BLEND_MODE_ALPHA_BLEND`` if it is supported. To enable reconstruction passthrough, you might add the following to the script attached to the root node of your main scene: + +.. code-block:: gdscript + + extends Node3D + + @onready var world_environment: WorldEnvironment = $WorldEnvironment + + func _enable_passthrough(enable: bool) -> void: + var openxr_interface: OpenXRInterface = XRServer.find_interface("OpenXR") + + # Enable passthrough if true and XR_ENV_BLEND_MODE_ALPHA_BLEND is supported. + # Otherwise, set environment to non-passthrough settings. + if enable and openxr_interface.get_supported_environment_blend_modes().has(XRInterface.XR_ENV_BLEND_MODE_ALPHA_BLEND): + get_viewport().transparent_bg = true + world_environment.environment.background_mode = Environment.BG_COLOR + world_environment.environment.background_color = Color(0.0, 0.0, 0.0, 0.0) + openxr_interface.environment_blend_mode = XRInterface.XR_ENV_BLEND_MODE_ALPHA_BLEND + else: + get_viewport().transparent_bg = false + world_environment.environment.background_mode = Environment.BG_SKY + openxr_interface.environment_blend_mode = XRInterface.XR_ENV_BLEND_MODE_OPAQUE + +Projected Passthrough +--------------------- + +.. image:: img/passthrough/projected_passthrough.jpg + +In this passthrough mode (sometimes referred to as geometry passthrough), passthrough is "projected" onto custom geometry, meaning passthrough imagery will only be visible in certain areas. +You might use this passthrough mode to show only specific objects in a user's physical environment, while rendering the rest in VR. + +Enabling projected passthrough only requires the addition of a visible :ref:`OpenXRFbPassthroughGeometry ` +node as a child of `XROrigin3D `_. Don't forget to assign the node's ``mesh`` property! + +.. image:: img/passthrough/node_tree_passthrough_geometry.png + +.. note:: + If the ``XRInterface.XR_ENV_BLEND_MODE_ALPHA_BLEND`` blend mode is set and supported on the headset, reconstruction passthrough will always take priority over projected passthrough. + +It's important to understand the :ref:`enable_hole_punch ` property on this node. With this property set to ``true``, +anything Godot renders behind the mesh will be removed from view. If set to ``false``, objects rendered behind the mesh will **still be visible in front of the passthrough**. +By default, ``enable_hole_punch`` is set to ``true``. + +.. figure:: img/passthrough/projected_passthrough_hole_punch.jpg + :align: center + + Hole-punched geometry hiding parts of controller models that are rendered behind the meshes. + +It's not possible to display hole-punched ``OpenXRFbPassthroughGeometry`` nodes simultaneously with nodes that have hole-punching disabled. +This is because hole-punching requires the viewport's ``transparent_bg`` property to equal ``false``. +If you want your passthrough geometry to have hole-punching disabled, you might use a startup script similar to the one in :ref:`reconstruction-passthrough` +**without** updating the ``environment_blend_mode``. + +Passthrough Filters +------------------- + +Meta headsets are capable of applying a variety of filters to the passthrough imagery, allowing you to modify the appearance of the user's physical environment. +There are a number of methods in :ref:`OpenXRFbPassthroughExtensionWrapper ` that allow you to access these features. +You can call these methods on the ``OpenXRFbPassthroughExtensionWrapper`` singleton, which can be obtained like so: + +.. code-block:: gdscript + + var fb_passthrough = Engine.get_singleton("OpenXRFbPassthroughExtensionWrapper") + +You can check what :ref:`PassthroughFilter ` is enabled by calling :ref:`get_current_passthrough_filter `. +By default, the filter mode will be set to ``PASSTHROUGH_FILTER_DISABLED``. + +To change the current passthrough filter, you can call one of the following functions: + +* :ref:`set_color_map ` +* :ref:`set_mono_map ` +* :ref:`set_brightness_contrast_saturation ` +* :ref:`set_color_lut ` +* :ref:`set_interpolated_color_lut ` + +The current passthrough filter can also be updated by calling :ref:`set_passthrough_filter `. +This function will most often be used to **disable** passthrough filters, as calling it with ``PASSTHROUGH_FILTER_DISABLED`` is the only way to return to that filter mode. +Swapping to any other filter mode using this function will require its corresponding setter function in the above list to have been called previously. + +Also note that there are two filter properties that can be set regardless of the current filter mode. +These can be set using the functions: + +* :ref:`set_edge_color ` +* :ref:`set_texture_opacity_factor ` + +.. figure:: img/passthrough/edge_color_passthrough_filter.jpg + :align: center + + Reconstruction passthrough after setting a red edge color. + +.. note:: + Values passed into ``set_edge_color()`` and ``set_texture_opacity_factor()`` will apply to **all filter modes**, even ``PASSTHROUGH_FILTER_DISABLED``. + +Color Map Filter +---------------- + +The :ref:`set_color_map ` method accepts a `Gradient `_ +resource and will set the filter mode to :ref:`PASSTHROUGH_FILTER_COLOR_MAP `. +This filter will map the input luminance value of the passthrough imagery to the corresponding RGBA value of the given gradient. + +.. image:: img/passthrough/color_map_passthrough_filter.jpg + +Here's what it might look like in the script and inspector to achieve this: + +.. code-block:: gdscript + + @export var color_map: Gradient + + ... + + fb_passthrough.set_color_map(color_map) + +.. image:: img/passthrough/passthrough_color_map_resource.png + +Mono Map Filter +--------------- + +The :ref:`set_mono_map ` method accepts a `Curve `_ +resource and will set the filter mode to :ref:`PASSTHROUGH_FILTER_MONO_MAP `. +This filter will map the input luminance value of the passthrough imagery with the corresponding grayscale value of the given curve. + +.. image:: img/passthrough/mono_map_passthrough_filter.jpg + +Here's what it might look like in the script and inspector to achieve this: + +.. code-block:: gdscript + + @export var mono_map: Gradient + + ... + + fb_passthrough.set_mono_map(mono_map) + +.. image:: img/passthrough/passthrough_mono_map_resource.png + +Brightness Contrast Saturation Filter +------------------------------------- + +The :ref:`set_brightness_contrast_saturation ` method accepts three ``float`` values +and will set the filter mode to :ref:`PASSTHROUGH_FILTER_BRIGHTNESS_CONTRAST_SATURATION `. +This filter will adjust the brightness, contrast, and saturation of the passthrough imagery. The valid ranges of each value are noted in the method description of :ref:`set_brightness_contrast_saturation `. + +.. image:: img/passthrough/bcs_passthrough_filter.jpg + +Here's what it might look like in the script to achieve this: + +.. code-block:: gdscript + + # Optionally allow the values to be edited in the inspector as a Vector3. + @export var brightness_contrast_saturation: Vector3 = Vector3(10.0, 0.5, 0.5) + + ... + + fb_passthrough.set_brightness_contrast_saturation(brightness_contrast_saturation.x, brightness_contrast_saturation.y, brightness_contrast_saturation.z) + +.. _color-lut-filter: +Color LUT Filter +---------------- + +.. note:: + `This article `_ by Meta provides a good introductory explanation on how to create a color LUT (Look Up Table / Look Up Texture). + +The :ref:`set_color_lut ` method accepts an :ref:`OpenXRMetaPassthroughColorLut ` resource +along with a ``float`` between ``0.0`` and ``1.0`` that is used to blend linearly between the original passthrough input colors and the mapped color LUT. +This method will set the filter mode to :ref:`PASSTHROUGH_FILTER_COLOR_MAP_LUT `. + +.. image:: img/passthrough/color_lut_passthrough_filter.jpg + +.. figure:: img/passthrough/passthrough_color_lut_resource.png + :align: center + + The color LUT used to achieve the above inverted color effect. + +To create an ``OpenXRMetaPassthroughColorLut`` resource, you'll have to pass an `Image `_ resource along with a :ref:`ColorLutChannels ` value +into the static ``OpenXRMetaPassthroughColorLut`` :ref:`create_from_image ` method, similar to the example below: + +.. code-block:: gdscript + + # Assign the color LUT image in the inspector + @onready var color_lut: Image + + ... + + var meta_color_lut: OpenXRMetaPassthroughColorLut = OpenXRMetaPassthroughColorLut.create_from_image(color_lut, OpenXRMetaPassthroughColorLut.COLOR_LUT_CHANNELS_RGB) + +Then, to fully apply the color LUT, we pass in ``meta_color_lut`` with a ``weight`` value of ``1.0`` to the setter function like so: + +.. code-block:: gdscript + + fb_passthrough.set_color_lut(1.0, meta_color_lut) + +.. note:: + You can check the maximum color LUT resolution supported by the headset at runtime using the :ref:`get_max_color_lut_resolution ` method. + +Lastly, if you want to smoothly interpolate between two given color LUT ``weight`` values over time, you can use a tween! The following example will interpolate the weight from ``0.0`` to ``1.0`` over a period of two seconds. + +.. code-block:: gdscript + + var tween = create_tween() + tween.tween_method(fb_passthrough.set_color_lut.bind(meta_color_lut), 0.0, 1.0, 2.0) + +Interpolated Color LUT Filter +----------------------------- + +This filter mode works the same way as the :ref:`color-lut-filter`, only it requires the use of two :ref:`OpenXRMetaPassthroughColorLut ` resources. +The :ref:`set_interpolated_color_lut ` method will accept those two resources, along with a ``float`` value between ``0.0`` and ``1.0`` +that will be used to blend linearly between the two color LUTs. Calling this function will set the filter mode to :ref:`PASSTHROUGH_FILTER_COLOR_MAP_INTERPOLATED_LUT `, +which will map the passthrough input to this interpolated color LUT. + +The main purpose of this filter mode is to smoothly transition between two distinct color LUTs. Let's assume we have two valid ``OpenXRMetaPassthroughColorLut`` resources named ``meta_color_lut`` and ``meta_color_lut2``. +If ``meta_color_lut`` is fully applied, we can smoothly transition to ``meta_color_lut2`` over a period of two seconds using a tween like so: + +.. code-block:: gdscript + + var tween = create_tween() + tween.tween_method(fb_passthrough.set_interpolated_color_lut.bind(meta_color_lut, meta_color_lut2), 0.0, 1.0, 2.0) diff --git a/docs/manual/meta/scene_manager.rst b/docs/manual/meta/scene_manager.rst index 2d3ebf4e..6037b5a3 100644 --- a/docs/manual/meta/scene_manager.rst +++ b/docs/manual/meta/scene_manager.rst @@ -9,7 +9,7 @@ Adding the Scene Manager Node To add the Scene Manager to your project, add the :ref:`OpenXRFbSceneManager ` as a child node of the XROrigin3D node. -.. image:: img/scene_tree.png +.. image:: img/scene_manager/scene_tree.png Let's go over the different options of the Scene Manager. The :ref:`Default Scene ` will be instantiated by the Scene Manager for any type of spatial entity (e.g. Walls, Doors, or the Global Mesh) and the ``setup_scene(entity: OpenXRFbSpatialEntity)`` method will be called on the created instance. The name of the method that will be called can be set using the :ref:`Scene Setup Method ` property. @@ -18,22 +18,22 @@ The :ref:`Auto Create ` option The **Scenes** section in the options allows you to individually assign a scene to a kind of spatial entity which can help in separating the logic. -.. image:: img/scene_manager_options.png +.. image:: img/scene_manager/scene_manager_options.png Next, head over to the Export Settings and make sure that in the **Meta XR Features** the ``Use Anchor API`` and ``Use Scene API`` options are enabled. -.. image:: img/export_settings.png +.. image:: img/scene_manager/export_settings.png Creating Spatial Entities ------------------------- Spatial Entities are a collective term for any object that has been mapped by the user in their space. These can be walls, doors, tables, plants, or many other things. -.. image:: img/spatial_entities.png +.. image:: img/scene_manager/spatial_entities.png First, create a new scene with the type of ``StaticBody3D`` and add a ``Label3D`` to it. The scene should now look like this: -.. image:: img/spatial_entity_scene.png +.. image:: img/scene_manager/spatial_entity_scene.png Next, add a new script with the following content: @@ -70,9 +70,9 @@ The Global Mesh is created by the headset when you scan your space with the dept |global_mesh| |global_mesh_wireframe| -.. |global_mesh| image:: img/global_mesh.png +.. |global_mesh| image:: img/scene_manager/global_mesh.png :width: 49% -.. |global_mesh_wireframe| image:: img/global_mesh_wireframe.png +.. |global_mesh_wireframe| image:: img/scene_manager/global_mesh_wireframe.png :width: 49% The process of creating the template scene for the global mesh is the same as for any other spatial entity.