Skip to content

Latest commit

 

History

History
219 lines (171 loc) · 15.4 KB

README.md

File metadata and controls

219 lines (171 loc) · 15.4 KB
Raw

README

Spectator View Rig

Overview

Spectator view uses an external camera rigidly mounted to a HoloLens to take high definition pictures and video of your holographic Unity project. Unlike Mixed Reality Capture (MRC), the Spectator View project allows for high resolution captures from a third person perspective Use this project to capture higher resolution and higher quality pictures for your store submissions, marketing videos, or for projecting a live view for all spectators to see.

DISCLAIMER: Video recording is in alpha and you might experience some latency between hologram position in video to real-life. Move the camera slower or target a stationary camera to mitigate this.

Material List

Note: The linked materials are not the only materials that will work, but are the ones that we used. Make sure the materials you get are compatible with your existing camera.

  • Camera with HDMI out and configurable output overlays.
    • See below for tested cameras.
  • Capture Card
    • See below for tested capture cards.
  • HDMI to Mini HDMI cable (or whatever HDMI form your camera supports)
  • Bracket to connect your HoloLens to a camera: Hardware\HoloLens_Mount.stp.
    • We recommend this to be professionally machined in aluminum: 3D printing might not achieve adequate tolerances to hold the device.
      • Search for machine shops in your area to get this bracket machined.
    • You do not have to use this bracket: Any means of rigidly attaching your HoloLens to your camera will work.
  • If using the above mount:
    • Adapter to connect the HoloLens mount to a DSLR hotshoe: Hardware\Mount_Adapter.stl. This file can be 3D printed. Holes were measured against the below bolts, you might need to alter the dimensions if using other bolts.
    • Hotshoe Fastener
    • 1/4-20" x 3/4" Bolts
    • 1/4-20" Nuts
    • Screw driver with the following heads (These links are examples, many screwdriver sets will have these attachments) :

Screw Drivers

Software

Dependencies

The following dependencies require a manual download and a Visual Studio user macro.

Dependencies

OpenCV 3.1

  • Download and install OpenCV 3.1 from here.
  • Extract OpenCV anywhere on your computer.
  • Update the OpenCV_vc14 user macro in dependencies.props with the corresponding path on your computer.
  • Restart Visual Studio if the Calibration or Compositor sln's are open.

DeckLink Capture Card

  • Download the DeckLink SDK from here.
    • Search for Desktop Video SDK in "Latest Downloads"
  • Extract the SDK anywhere on your computer.
  • Update the DeckLink_inc user macro in dependencies.props with the corresponding path on your computer.
  • Restart Visual Studio if the Calibration or Compositor sln's are open.

Elgato Capture Card

  • Download the Elgato DirectShow filter headers from here.
  • Clone or extract anywhere on your computer.
  • Update the Elgato_Filter user macro in dependencies.props with the corresponding path on your computer.
  • Restart Visual Studio if the Calibration or Compositor sln's are open.

Canon SDK (Optional)

  • Request access to the Canon SDK from here.
  • Extract anywhere on your computer.
  • Update the Canon_SDK user macro in dependencies.props with the corresponding path on your computer.
  • NOTE: When targetting x64 Unity, use EDSDK_64. When targetting x86 Unity, use EDSDK.
  • Restart Visual Studio if the Calibration or Compositor sln's are open.

Included Projects

Each project has its own README with additional overview and setup instructions. See Documentation links below.

Calibration

Calibrates the HoloLens to the external camera. Uses a checkerboard pattern to find the rotation and translation from the HoloLens to the external camera. This process also finds the vertical field of view of the attached camera so holograms can be rendered at the correct projection to the camera.

Compositor

Unity Editor window that renders holograms from Unity over a color frame from a capture card. Outputs photo and video of the scene. This must be built and copied to your Unity project using CopyDLL.cmd.

Samples\SharedHolograms

Sample demonstrating how to add the HolographicCameraRig to an app that is already a shared experience. In this case, it is the Holographic Academy Holograms 240 course.

CopyDLL.cmd

If you have changed any #define in CompositorShared.h, run this cmd to copy DLL's that depend on those definitions to your Unity projects. Include the Assets path to your own project to copy there.

Camera Setup

If using a camera with a capture card, you must configure the camera to output correctly:

  • Change your camera to Video Mode. This will allow the output image to fill the whole buffer, rather than letterboxing to the picture aspect ratio.
  • Find your camera's HDMI settings - enable "dual monitor" or "mirroring".
  • Set the resolution to 1080P
  • Turn off "Live view on screen display" so any screen overlays on the live view are not shown in the composite.
  • Turn on your camera's live view.
  • If using the Canon SDK and would like to use a flash unit, disable "Silent LV Shoot"
  • Plug an hdmi cable from the camera's hdmi port to the hdmi in port of the capture card.
    • This is the outer port on the Blackmagic capture card.
  • Optionally plug an hdmi cable from the hdmi out port to a television or preview monitor.
    • This is the center port on the Blackmagic capture card.

Note: if you would like to capture images or video at a resolution other than 1080P (eg: Photo Mode, 720P, or 4K) there will be a small code change mentioned in the individual projects' README's.

Spectator View Rig Setup

This section assumes you are using the included HoloLens bracket. Alternative mounting techniques can be used instead.

Assembly

  • Use a T7 screwdriver to remove the headband from the HoloLens.
    • Once screws are loose, poke them out with a paperclip from the other side.
  • Remove the screw cap on the inside front of the HoloLens visor, you can do this with a small flat head screwdriver.

HoloLens with no headband

  • Use a T15 screwdriver to remove the small torx bolts from the HoloLens bracket to remove the U and Hook-shaped attachments.
  • Place the HoloLens on the bracket, lining up the exposed hole on the inside of the visor with the extrusion on the front of the bracket.
    • HoloLens arms should be kept in place by the pins on the bottom of the bracket.
  • Reattach the U and Hook-shaped attachments to secure the HoloLens to the bracket.

Bracket

  • Attach the hotshoe fastener to the hotshoe of your camera.
  • Attach the mount adapter to the hotshoe fastener.

Adapter

  • Rotate the adapter so the narrow side is facing forward and parallel to the camera's lens.
  • Secure the adapter in place with a 1/4" nut using the 7/16 nut driver.
  • Position the bracket against the adapter so the front of the HoloLens' visor is as close as possible to the front of the camera's lens.
  • Attach the bracket with 4 1/4" nuts and bolts using the 7/16 nut driver.

Alternate Spectator View Rig Setup

The spectator view rig assembly can be made without any custom hardware.

Alternate Assembly

Material List

Assembly

  • Attach the hotshoe fastener to the camera's hotshoe using one of the included nuts.
  • Place a washer on the hotshoe fastener above the nut.
  • Attach the dual flash bracket to the hotshoe fastener.
    • The washer should guarantee that the bracket is perpendicular to the lens.
  • Remove the included thumbscrews from the dual flash bracket.
  • Attach the clamps to each side of the dual flash bracket with the 6/32 nuts and bolts.
    • Ensure the arms of the HoloLens will be able to fit in the clamps with the visor centered on the camera's lens.
  • Place each HoloLens arm in the open clamps.
  • Place some cardboard or stiff foam in between the headband and the arms to act as a buffer against the clamp's tension.
  • Without overtightening, tighten the clamps to keep the HoloLens in place.

Recommended PC Specs

Any PC that can run the HoloLens emulator can run spectator view.

Tested Cameras

Tested Capture Cards

NOTE: The default capture card will be a blackmagic capture card, if using a different card, you will need to update Compositor\SharedHeaders\CompositorShared.h:

  • USE_DECKLINK is TRUE if using a Blackmagic card.
  • USE_ELGATO is TRUE if using an Elgato card.
  • USE_OPENCV is TRUE to fallback to any other camera.

Each capture card has required software dependencies including runtimes and SDK's. See the Software section above.

Adding Spectator View Support to Your Unity App

  • If your application does not use the HoloToolkit, you must first add the HoloToolkit.
    • This can be done from the UnityPackage directory for a minimum set of the HoloToolkit for spectator view compatibility.
    • Find the largest number subdirectory for the latest official release.
    • Import the HoloToolkit_Min_%number%.unitypackage
  • Import the SpectatorView_%number%.unitypackage to add the spectator view code to your project.
  • Unzip the External_%number%.zip archive adjacent to your project's Assets directory.
    • Ensure the unzipped directory is titled "External" and does not have another "External" subdirectory.
  • Add the "Addons\HolographicCameraRig\Prefabs\SpectatorViewManager" prefab to your Hierarchy.
  • Build "Compositor\Compositor.sln" and run CopyDLL.cmd with the Assets directory of your project as the only parameter.
  • For more information, read the sample README.

NOTE: You can alternatively use the latest HoloToolkit from the github repository - The sample project is kept up to date with breaking changes before official drops are made, but this code is constantly changing so additional work may be required to ensure compatibility. If you are using this HoloToolkit, you can use the spectator view UnityPackage in "UnityPackage\HoloToolkit_master".

Documentation