Introduction: The Ricoh Theta 360 camera can describe as a portable device on steroid, put it in the right user hands then endless applications areas couple with extreme exploration innovative ideas and knowledge that have real-world cases built into them. The Theta 360 camera can be employed in many sectors ranging from retails, transport, government, research, construction, marketing, Internet of Things, machine learning to mention but few of those endless areas of its applications.
But it need to be seen, handled and used for its versatile features to be appreciated and put into practical real-world usages in many of those sectors.
This is where we believe that a software framework that embraces all Theta-360 camera and bring into one-place is an important stepping stone in order to expand it's user base and adoption World-wide.
Proposed framework will also facilitate software porting into Ricoh Theta-360 camera and extends its current plug-in module beyond its current status.
For an End-to-End, it has to embrace low-code if it has to be in the hand of builders, retailers, transporter, haulage trackers, products monitor, home users and other non-programmers and non-developers.
We will develop drag and drop low-code environment that could be used to create quick Apps for Ricoh Theta devices. It will features "Rapid-IDE" that even run inside Theta device itself and be self contained and could be update as need e.g. OTA(Over the Air) or via download/update features. It will be possible to create Android Apps or Apps that run on Theta devices.
What we set out to achieve but in no preference order:
- Setup a Github repository that will house our resources for Theta V and Z1: So for now I am using: https://github.com/sanyaade-teachings
- A mini IDE environment to help develop application around Theta devices. A port of some framework around Cordova seems that right to go on this path.
- On the customisation or personalising the Theta devices there are few options available here: 1.) Custom ROM development (based on Android AOSP) 2.) Yocto build tool or Buildroot. But whichever way is embarked on here, there will be need for 1.) Android source code (for Android phone) or iOS for iphone.
We will need to tackle the two issues raised above as they are fundametal to whatever we set to achieve around the Theta V and Z1 devices.
I started by porting some IDE to work with Theta V and Z1 devices:
- Evothings framework
- ionic-creator
- possibly Wakanda framework
- an enhanced WebRTC
- Machine Learning for Embedded Environment
- Cordova/PhoneGap
All of these framework are based around Cordova usng javascript for native platform and were also cross-platforms. We are going Low-Code as this will attract hardware Hackers/Makers/Developers including who cannot or does not get into code part but will like to create some Apps quickly for the Theta V and Z1 or to integrate their expansion modules with Theta V and Z1.
I do both the hardware side and the software side together in day to day routines. Both are well balanced on my daily agenda.
We will follow what are listed on the Theta360 developer guide at: http://theta360.guide/plugin/ covering those six areas that were listed as much as possible:
PART 1: A Rapid Prototyping IDE for Theta V/Z1
I will stack a modified version of Evothings framework that I am porting across to Theta 360 platform. I embark on porting this across to enable people to quick prototype, test and create mobile applications and IoT Apps that will run inside Theta devices or interact with. Evothings is based on Cordova and can extend with Ionic, TypeScript, C/C++, Java, etc... but for now we keep things down to html, CSS and Javascript (Cordova platform).
We need to setup and install dependencies for building our software:
First we need to install Nodejs which is a server side JavaScript: The following steps show how to get Nodejs setup for our need:
cd /tmp$ wget http://nodejs.org/dist/v10.15.3/node-v10.15.3-linux-x64.tar.gz
$ tar xvfz node-v10.15.3-linux-x64.tar.gz
$ mkdir -p /usr/local/nodejs
$ mv node-v10.15.3-linux-x64/* /usr/local/nodejs
For MS-Windows setup, see: https://nodejs.org/en/download/
Step 1: The Theta-Rapid Studo
In our first step we will build modify and build a version Theta-Rapid Studio source which is based and derived from Evothings code from Github:
Open the terminal, type:
git clone https://github.com/evothings/evothings-studio.git thetaRapidStudio
The above command will download the source code from Github and create a folder named thetaRapidStudio. Next will modify some files and make changes to configurations files, parameters, move and add directories to reflect the new Theta-RapidStudio environment.
If everything is okay, you have a directory named thetaRapidQ as shown below:
Next change into that directory, type:
cd thetaRapidQ and press enter
Homework@Homework-PC MINGW64 /c/wkdir-1
$ cd thetaRapidQ
Homework@Homework-PC MINGW64 /c/wkdir-1/thetaRapidQ (master)
$ ls
app/ build/ build.sh* jsconfig.json LICENSE licenses/ package.json
Homework@Homework-PC MINGW64 /c/wkdir-1/thetaRapidQ (master)
Next type the following commands to get the package built:
npm install
npm install app
npm start
The above commands will:
- npm install: this command will download and get all needed npm dependencies for development and building.
npm installapp
: will install all needed npm dependencies for our application.
npm start:
Will load and run the application ready for our use so that we start to create mobile apps for iOS and Android.
There we are, a newly built initial state of our Theta-Rapid Studio. The IDE is up and running.
Next stage is to customise it so that it can work very well and fit soundly meeting the what are needed for developing Apps for Theta V and Z1. The good news is that, it can also create Apps that will run on old model of Theta camera so don't throw away those one yet. Of course old Theta-360 camera may not have those newly features that were incorporated into Theta V and Z1
So where do we start to make those changes?:
Well let look at the folder layout of our studio first:
From the above screenshot, the folders are explained below:
App: as the name indicated, this contains all our Apps software. This is where all our App created will be stored both current and new ones will be in that folder.
Inside the App folder:
Important parts of the App folder are:
- libs: contains bootstrap lib the we employed for our application. Bootstrap is a light library suitable for single-page App development.
- codemirror: this is our in App code/text editor. By incorporating a code/text editor directly into our studio means we can reduce dependencies as much as possible. This is important for an End-to-End application. Users can edit directly without having to installed an editor.
- server: Yes! Our studio houses its own web-server. The logic behind this is to be able to do client-server communication with real-live update of applications as they are being created or modified in suit. This is also very important as it we permit Theta V/Z1 running studio to communicate bi-directional with another App(s) running in another device or in vice versa the studio running in another device say ESP32, an Arduino, Raspberry Pi 3 or in mesh environment can also connect, communicate and exchange data with an App running inside Theta V or Z1. flexibility and portability are the keys important here. The Server setting can be changed and configure to meet various use cases and scenario. as we shall in some of our demo Apps later. With a little knowledge of JavaScript, everything inside the server including setting, URL, port, etc.. can modified;
important areas of the server setting that you may want to change are things like:
Server Port: systemSetting('WebServerPort', 4042)
Discovery Setting: systemSetting('ServerDiscoveryEnabled', true)
systemSetting('ServerDiscoveryPort', 4088)
For Live Update/Inter-process communication: you can modified the address of the synchronising/reloading server.
systemSetting('ReloadServerAddress', 'https://app.local')
var defaultValue = 'localhost'
Also many other parameters including Application name, Author, etc.. can be chnaged. See the code section for the complete server setting snippet or at the Github repository for this project:- https://github.com/sanyaade-teachings/theta-rapid-studio
settings: The setting folder contains the studio configuration file (settings.js) that is loaded when the application is started.
- ui: The ui folder contains user interface elements like themes, presentations, look and feel. These are again editable and can be customised/personalised to various individual needs. You only need HTML, CSS and JavaScript skills:
- injectables: As you can see from the above, I stared this folder. This folder contains interest JavaScript code that can be modified to change the behaviour of our studio or add more features to expand it beyond its current capabilities with third party libraries.
- plugins: This folder contains various plugin like cordova-plugins and other modules that we will be incorporating to expand the studio functionality:
Theta V, Z1 and other interface modules that we will like to bring in be stored here. Think about this place as directory for Foreign Function Interface part of our software
- node_modules: unless you are not familar with Node.js packages. Our studio software as I said at the begin is a Node.js application. Well it is an hybrid of Node.js (server side javaScript) and Cordova (Application side HTML, CSS, JavaScript) combine to give us cross-platform native mobile app creation environment. You might have noticed from the plugins folder contents that we also make use of other technologies like coffeeScript. TypeScript is also on our radar since we will be doing JavaScript Embedded Systems development working with devices like ESP32, Arduino, Occulus Rift, Raspberry Pi 3, BeagleBone Black, Silego GreenPak, IOIO-OTG and more... A lot of changes has been brought into how we use JavaScript nowadays. Web Applications and Desktop Applications are merging into one sooner there will be no more differences once the operating systems platforms had been abstracted very well beyond HAL :) The node_modules folder contains all our node.js dependencies modules that our Theta-Rapid-Studio rely on.
- build: These is where you will put all the program icons that will be pulled in during studio compilation process. It is important to keep the icon size reasonable at 32x32 or 64x64 pixel. Employ png if you want transparencies and svg for scaling up and down without degradable quality.
Note: there are alternatives implementations out there but size, portability and customisation has been my key drive into adapting Evothings as jump-start and as startup. It is easy to understand, very well laid out codes and modules can be expanded, removed or changed to suit your needs. There is a Gitter forum to ask questions. I have been play with for few year and used it for other needs.
Studio Viewer: So far we have got our App development studio for Theta devices. Is that all that we need? Yes! and No!. I start with the No! part first, you need an App Viewer during the development sessions. The viewer is used for live update or what we called continuous/incremental development. It lets you see changes you have made to your App instantly. Why do we need this? I bet you might asked? Do you forget? And I hope that you know that Theta V, Z1 and other Theta devices do not have any output screens at all. Think about Theta device as mobile-phone hardware, they have everything your phone has including special 360 degree view camera and snap-dragon features but unlike your mobile-phones, the Theta device do not posses a display screen. No touch screen, output screen, feedback screen as part of the device so the only to get output or display information is via a cable to your computers, display monitors or your TVs. Hey!
During my preliminary investigation, I thought about being in rural areas, mountain climbers, oceanic scenery captures, robotics application, factory or home automation uses. Having to carry around or buying additional screen may be putting people off but if you already have a mobile-phone that you can just use the studio as server and your mobile-phone as client or vice-versa and get things going. No hassle right!
On the Yes! part, the viewer is not need once you created your App either for Android or iOS platform. You App will run independent of the studio viewer App. The Viewer App is very small less than 3-MB for an Android and both you App and the Viewer can co-exist without any conflict.
To build a version for your Theta360 Camera, following the outline steps below:
1.) First go to: https://developer.android.com/studio#downloads and download, install and setup Android SDK (Command Line Tool) for your operating systems platform. Note you don't need the Android Developer Studio because we have our own made for Theta Camera you just need the SDK with command line interface. But if you like you can install the full Android Studio, if you want to learn more about or kotlin language (https://kotlinlang.org/) that is replacing the Java part of Android. Good move by Google and same with Apple moves by the Swift language (https://swift.org/) replacing Objective-C. Both of these languages are 100% open source and cross-platform. They get things done fast and large reduction of code sizes. I will advise you to take a look at them on their Github, most especially to learn how commercial software tools are made.Look at their lexers, parsers and code generator before they get bloated up like their predecessors (e.g. Java, Object-C, C#, QT, etc..)
Apart from the Android SDK, we will need Cordova setup. If you are running all these directly on Theta V or Z1 then you need to download the Android version of Cordova. Lets go:
First install Cordova globally on your device:
npm install -g cordova
if
everything goes very well then should something similar to the above screenshot.
Next navigate to the thetaStudio-viewer source directory:
cd thetaStudio-viewer
Add your platforms: lets focus on Android for now as mention inn the Theta Developer Guide website:
cordova platform add android
Then build using Cordova:
cordova build android
Or for release:
cordova build --release android
be
patient a little bit as things need to be downloaded from the internet, may be in your case through WiFi of your Theta V or Z1 but they will get installed
Infrastructure Layout:
What we trying to achieve can be as shown below but things may change along paths and directions. PhoneGap/Cordova has a Developer App that I am examining now as I write the project documentation and there is a plan to bring those features into Theta V and Z1 platforms. See below for our layout snapshot:
You can see how and where we can get plugins done for Theta V/Z1 through many routes.
Below is also the layout of how our Apps and Theta devices will establish device to device communication. Communication routing can be via Edge computing, via Cloud or on a Peer-2-Peer basis. Good on flexibility!
For the inter-process communication, RestFul, RPC, Serial communications and more... The diagram below shows how plan to get things done.
Note: Please note that most of the layout and ideas are following Evothings and are borrowed from them since our framework is using that as basis to get things running initially. We have to start from somewhere and then move on... Reinventing from scratch is very costly nowadays due to fast rate at which technologies are moving...
Our next step, well obviously is to build some real live demos that proof everything we have outline and done so far. To out it in picture below is what we have lying around to showcase for this project:
We will demo how we interact and exchange data, connectivity with each of items in the picture above. We will be using Photosynth and OpenSynther for doing all sort of things from Panorama, 360 degree walk-through and various. Our WebRTC already has computer vision, incorporate SOD for machine-leaning for embedded systems (uTensorFlow)...
This project is going and an ongoing
For resources used and more, please see the following Github repository:
[Theta-Rapid Studio]: https://github.com/sanyaade-teachings/thetaRapidStudio
[thetaStudio-viewer]: https://github.com/sanyaade-teachings/thetaStudio-viewer
[thetaStudioReload]: https://github.com/sanyaade-teachings/thetaStudioReload
Please join us as we moved to the development last phase.
God blesses!!!
Best reagards,
Sanyaade
Comments
Please log in or sign up to comment.