Stereoscopic (3D) Streaming Camera and Viewer - Part 2

In version 1 of this project, I created a Tripod-mountable stereoscopic 3D Camera that transmits a stereoscopic video stream over HTTP so it can be viewed on a stereoscopic viewer like Google Cardboard. I used two Raspberry Pi Zeros in that setup, each with its own camera.

In this updated project, I'm utilizing a Raspberry Pi 5 which has two camera slots and can thus accommodate both cameras on the single board.

The code has also been updated to utilize the new PiCamera2 python library. Version 1 utilized PiCamera, as the latest Raspberry Pi OS still didn't have python support for libcamera. The updated code supports both setups - using 2 boards, 2 cameras setup with Raspberry Pi Zeros or the 1 board, 2 camera setup with the Raspberry Pi 5. It also utilizes Javascript to prevent screen auto-lock.

In this project, I'll focus mainly on the Raspberry Pi 5 setup. You can refer to the previous project for specific details on the Raspberry Pi Zero setup, including mounting them on the enclosure and sharing the power rails, so both can be powered from a single power adapter.

Stereoscopic 3D images/video attempt to replicate what we see with our own eyes. Your eyes are approximately 2.5 inches or about 6cm (60mm) apart (also called interocular distance). As a result, each eye sees the image from slightly different perspectives and angles. The brain combines these two images into a seamless image and also uses the information to gauge distance and depth. This is also called binocular vision.

We can replicate this with a 3D Stereoscopic Camera, by having two cameras that are separated by roughly the interocular distance. These two images (left image and right image) are then presented individually to each eye using a stereoscopic viewer like Google Cardboard. The brain then combines these images and processes the subtle differences in perspective and angles to create depth perception, or a 3D image!

In this project, I built a stereoscopic 3D camera using a Raspberry Pi 5 and two Pi Camera 2s, housed in a ProtoStax Enclosure for Raspberry Pi 3/4/5.

I also use the ProtoStax Stereoscopic 3D Camera Kit for Raspberry Pi Camera - 60mm Stereo Base which allows you to mount two Pi Camera 2 situated, you guessed it 60mm apart – our interocular distance (also called interaxial distance or stereo base when referring to camera setups)! It also comes with a tripod mount piece that makes the whole enclosure tripod-mountable.

We'll use Python to stream the camera outputs from the two cameras into a composite left-eye/right-eye browser-based video stream. You can then open this stereo video stream up on a browser on your mobile phone and place it in a stereoscopic viewer like Google Cardboard, and voilà! - we have our 3D stereoscopic Streaming Camera and Viewer!

Now you can give a 3D tour of your house to your Nana or Abuelo without having to make them climb the stairs or leave the comfort of their chair, as you go and operate the camera for them to view remotely!

Or have silly 3D games where you show your friends or family themselves as they view the world you present through the Google Cardboard - turn the camera and point it at them, and watch their reaction as they see themselves in 3D! Quite trippy!

Disclaimer - People with motion sickness can get a bit queasy when viewing VR headsets or stereoscopic cameras. If you start to feel queasy, just ease off the stereoscopic camera viewing! 😊

Another Disclaimer - the two cameras are NOT synchronized here, and this project is not intended for hardcore stereoscopic video use. For that purpose, you can use additional hardware such as Arducam's Synchronized Stereo Camera HAT for Raspberry Pi (or other similar products from Arducam) . The ProtoStax enclosure has enough room for a HAT, so you can definitely upgrade your project to use hardware synchronized cameras. You can also try to use software techniques to improve performance and improve synchronization. These are mentioned in the "Going Forward" section on how you can improve upon this project. However, that being said, the Raspberry Pi 5 is a lot more powerful than two separate streams from two Raspberry Pi Zeros, and offers decent performance even without hardware synchronization. You don't need any other hardware and can start playing around and experimenting, and the setup always allows you to upgrade to hardware synchronization if you so desire!

Ok, let's get started!

Step 1: Prepare your Raspberry Pi

If you are starting afresh, you need to first configure the SD card for your Raspberry Pi 5 with the Raspberry Pi OS. The Raspberry Pi Imager tool allows you to do some configurations ahead of time when you create your boot image.

Since we'll be using the Raspberry Pi in a headless setup (not connected to a monitor or TV), you'll need to enable SSH on them. We'll also want them to connect to the Wireless network directly on bootup. We can provide the WIFI network name and password credentials using the Raspberry Pi Imager tool setup.

Once we have done this, we can do the rest of the setup later, after we boot up our Stereoscopic Streaming Camera hardware setup. We can just SSH into the Raspberry Pi and finish the rest up the setup, including getting the source code and running the programs.

Step 2: Set up your hardware

Start by mounting the Raspberry Pi 5 on the ProtoStax Enclosure for Raspberry Pi 5 / 4 / 3 Model B/B+.

Next up, mount the Raspberry Pi Cameras to the Top Plate of the ProtoStax Stereoscopic 3D Camera Kit for Raspberry Pi Camera.

Connect the camera wires to the Raspberry Pi 5. Alternately, you can connect the cameras to the Raspberry Pi 5 before you mount the cameras to the Top Plate! Whatever you find easier!

The camera slot closest to the Ethernet port is Camera 0. The slot next to it (closer to the HDMI ports) is Camera 1. In my setup, I've connected the right camera to slot 0 and the left camera to slot 1. If you interchange them, just update the code appropriately.

Add the side walls (including the one/two Tripod Mount side walls from the ProtoStax Stereoscopic 3D Camera Kit for Raspberry Pi Camera), and close everything up.

Here is the final completed stereo camera setup, with an attached Tripod mount adapter plate (this will totally depend on the type of tripod you have!).

Step 3: Configure the Raspberry Pi

With the latest Raspberry Pi and Raspberry Pi OS, you don't need to do a lot of extra configuration. Earlier, you would need to enable the camera interface using raspi-config. Now, when cameras are connected to the interfaces, they'll automatically get enabled.

Step 4: Configure the Software

You can download the software from GitHub. Alternately, you can install it using git

sudo apt install git
git clone https://github.com/protostax/ProtoStax_StereoscopicCameraPi.git 
cd ProtoStax_StereoscopicCameraPi 
python3 stream.py

The program will expose two MJPG video streams of the cameras via a simple webserver, as:

http://raspberrypi5.local:8000/rightstream.mjpg and http://raspberrypi5.local:8000/leftstream.mjpg

(* replace raspberrypi5 above with the hostname you chose for your pi)

The program also has a home page (index.html) that will expose both video streams as a stereoscopic image (side-by-side), as

http://raspberrypi5.local:8000/

You can open the link on your mobile phone browser (just make sure your mobile phone is on the same network as the Raspberry Pi) and place the phone in a 3D stereoscopic viewer like Google Cardboard (or similar viewer) to get a 3D stereoscopic streaming camera on your viewer.

The viewer can thus see everything in 3D. The camera operator can move around to give the viewer a 3D tour from the comfort of their couch!

Step 5: Focusing the camera and other tips

For optimal performance, you can focus the lens of the Raspberry Pi Camera 2 for a crisper image. You can use Lens Adjustment Tool for Raspberry Pi Camera to easily adjust the focus.

For best streaming performance, make sure you don't have too many other programs or services running on your Raspberry Pi. If you're starting off with a clean slate with your SD card images, this should not be a problem. However, if you're starting with existing an Raspberry Pi that's already configured, do ensure that you've stopped and disabled extra services, so streaming performance is not affected.

When you open the stereoscopic streaming link on the browser on your mobile phone, it is best if you hide the toolbar, so extraneous elements are minimized in the viewer. You will need to adjust the phone laterally in the viewer so that the image appears as a single smooth one that appears 3D (and there are no blurry artifacts).

The updated code in this project also prevents screenlock from kicking in after a while, which would happen if the phone is placed in the Google Cardboard Viewer - just click on the "Wake lock is disabled" button to enable it! The background changes to black when it is enabled.

See it in action!

Code Dive

The code for this is based on the Picamera library's Advanced Recipe for Web Streaming of video, which starts recording the camera's video stream to a buffer and then feeds the frames via a http request handler. It has been updated to utilize PiCamera2 instead on the latest versions of Raspberry Pi OS. The web server is started on port 8000. You can see each individual mjpg stream of each camera using raspberrypi5.local:8000/leftstream.mjpg or raspberrypi5.local:8000/rightstream.mjpg

This simple web server will additionally configure both left and right mjpg streams into a single page with split views using some HTML and CSS and provides that in response to the request for the home page (raspberypi5.local:8000/ or the index.html page (raspberrypi5.local:8000/index.html ). Thus opening up the home page will provide a stereoscopic video stream.

Here is the HTML and CSS used to format the page:

<html>
  <head>
    <link rel="shortcut icon" href="https://www.protostax.com/cdn/shop/files/Sri
dhar_rajagop.png?crop=center&height=32&v=1613523611&width=32" type="image/x-icon
">
    <style>
      .viewfinder {
        display: flex;
        flex-wrap: wrap;
      }
      .viewfinder-item {
        height: 50%;
        width:  50%;
        object-fit: contain;
      }
    </style>
    <script type="text/javascript" src="https://richtr.github.io/NoSleep.js/dist/NoSleep.min.js"></script>

  </head>
  <body>
    <input type="button" id="toggle" value="Wake Lock is disabled" />

    <script>
      var noSleep = new NoSleep();

      var wakeLockEnabled = false;
      var toggleEl = document.querySelector("#toggle");
      toggleEl.addEventListener('click', function() {
        if (!wakeLockEnabled) {
          noSleep.enable(); // keep the screen on!
          wakeLockEnabled = true;
          toggleEl.value = "Wake Lock is enabled";
          document.body.style.backgroundColor = "black";
        } else {
          noSleep.disable(); // let the screen turn off.
          wakeLockEnabled = false;
          toggleEl.value = "Wake Lock is disabled";
          document.body.style.backgroundColor = "";
        }
      }, false);
    </script>
    <div class="viewfinder">
      <img class="viewfinder-item" src="leftstream.mjpg" />
      <img class="viewfinder-item" src="rightstream.mjpg" />
    </div>
  </body>
</html>

It utilizes the NoSleep.js javascript library from richtr to prevent the mobile phone screen from autolocking after a period of inactivity, which will happen when the mobile phone is in the Google Cardboard Viewer. In version 1, I recommeded you to turn off Auto-lock on your phone using its settings. This updated code makes it more convenient and easier to use.

The code also has another HTML/CSS setup for the 2 camera, 2 Pi Zero setup that combines streams from a leftcam.local and rightcam.local Pis (update the names to match your setup).

In order for the code to run automatically on bootup, you will want to run it as a service. The GitHub page's README has instructions on how to do that, including the configuration file for the service.

Going Forward:

As always, I like to end by talking about how you can take this project forward with your own additions! Once you get comfortable playing around with the code sample and understanding the code, it is always nice to try to extend your learning by doing more. Here are some suggestions:

• Run the python streaming program as a service on the Pi, so it gets started automatically when the Pi boots up. The GitHub link has instructions on how you can do that.
• We currently stream the video using the MJPEG format. Look into streaming using MPEG1 or other formats which may give better performance
• Add the ability to take a stereoscopic picture - add a button to the Raspberry Pi GPIO and use that trigger taking left and right photos from the two cameras and stitching them together as a stereoscopic duo
• The two cameras are not synchronized, so if you have video with a lot of motion, the inherent lag between the two camera feeds may end up making you a bit queasy if you are prone to motion sickness. You can use hardware camera synchronization with a product like Arducam's Synchronized Stereo Camera HAT for Raspberry Pi (or other products they offer) and make the performance better. The ProtoStax enclosure has enough room for a HAT inside, so you'll be able to expand and improve upon the project. You can also try to use software to improve the synchronization between the two feeds. You can use the PiShot project by some MIT students as inspiration and a basis - https://github.com/revalo/pishot - rock on!

Can you think of other ways to extend this project? Share it with us in the comments below! 😊 Also do share photos/videos of your creation on your social media platform of choice - just tag it #ProtoStax and tag us @protostax so we can find you!

Happy Making! 😊Sridhar