Time Leap Cam - Time-Lapse Cam and Monitoring by OpenAI

About This Camera

This camera, named "TIME LEAP CAM, " is designed for long-interval time-lapse photography and extended monitoring. The name reflects its ability to "leap" through time by capturing images over long periods. While there are many cameras with monitoring and time-lapse capabilities, none can adapt to various conditions and make intelligent judgments based on those conditions.

The inspiration for this camera came from a personal need. My elderly father, who is now safely residing in a care facility, had instances where he fell or was in an abnormal state at home. I wanted to receive notifications in such cases, but constantly monitoring a camera feed was impractical. Often, by the time an incident was noticed, it was already too late. Recognizing that many others might face similar situations, I decided to develop this camera.

Coincidentally, OpenAI released GPT-4 with Vision capabilities around the same time. I explored the possibility of sending captured images to OpenAI for content analysis. After confirming that images could be sent from a PC and analyzed, I decided to integrate this functionality into the camera.

One challenge was the need for communication to send images. To keep the camera power-efficient and cost-effective, I chose the ESP32S3 SoC, which supports WiFi communication and HTTPS (SSL) protocols. The camera uses WiFi to connect to the internet and employs OpenAI's GPT-4o to analyze the images. If certain conditions are met, the camera sends notifications via LINE. For example, it can notify if a person has fallen if a pet's food bowl is empty, if plants appear unhealthy, if a window is open, or if it looks like it might rain. This eliminates the need for constant human monitoring. The LINE notifications include relevant images, and the conditions for notifications can be easily set using natural language through a configuration screen.

The camera is battery-powered and enters DeepSleep mode when not capturing images to conserve power. Captured images are stored on an internal 64GB eMMC. There are 12 folders available for organizing the stored images. The camera is designed to be water-resistant for outdoor use, though it is not waterproof. Charging is done via a Qi wireless charger, and settings can be configured through a web browser on a smartphone or PC over WiFi. If the camera is in sleep mode, touching a pad on the back activates the web server for configuration access. The camera automatically returns to sleep mode after 5 minutes of inactivity.

To use the image analysis and LINE notification features, an OpenAI API key, CloudFlare Images service access ID and API key, and LINE Messaging API account ID and access token are required. Internet communication via WiFi is also necessary. For time-lapse photography only, WiFi communication is not needed.

Service IDs and access keys can be obtained from the respective service websites:

• OpenAI API: https://openai.com/index/openai-api/
• CloudFlare Images: https://developers.cloudflare.com/images/about
• LINE Messaging API: https://developers.line.biz/en/docs/messaging-api/

Features of This Camera

• Auto-focus
• Long-interval shooting
• Power-saving DeepSleep mode (less than 440uA during sleep, 1.5mW)
• Qi wireless charging with automatic stop when fully charged
• Water resistance
• Image analysis and LINE notifications using OpenAI GPT-4o/GPT-4o mini/GPT-4-Turbo models
• Natural language notification condition settings (multi-language input supported)
• Touch-activated web server and automatic sleep mode
• Time synchronization via NTP (automatic adjustment when WiFi is connected) and RTC for timekeeping

Specifications of This Camera

• Resolution: Up to 2592x1944 (QSXGA)
• Sensor: OV5640 5MP sensor
• SoC Module: ESP32-S3-WROOM-1-N16R8 (240MHz, 512KB SRAM, 16MB Flash, 8MB PSRAM)
• Battery: 3.7V LiPo 1100mAh
• Storage: 64GB eMMC
• Communication: WiFi 2.4GHz
• Charging: Qi wireless charging
• Dimensions: 67(W) x 78(H) x 66(D) mm

Notes

• Water Immersion: The camera is not designed to withstand full submersion in water.
• API Keys Required: To use the image recognition and notification features, obtain API keys from OpenAI, CloudFlare Images, and LINE Messaging API.
• Internet Connection: Required for using advanced features like image recognition and notifications. Time-lapse functionality is available without WiFi.

XIAO ESP32S3 Sense Version Released

https://github.com/hnz1102/timeleapcam/tree/XIAO_ESP32S3_SENSE

You can use XIAO ESP32S3 Sense Board, too. However, There are a few limitations. Please refer to github repository.

For TimeLeap Cam board Update.

Update v0.3.3

• Improved the eMMC writing speed. The eMMC writing speed is improved by background writing. In the HD resolution, the writing speed is improved 28fps to 30fps. In the Full HD resolution, the writing speed is improved 6fps to 8fps.
• JPEG quality can be set from 4 to 40. The default is 12. This value is high quality to set the value to 4. In the HD resolution, JPEG quality is set to less than 12, the writing speed is slower than 30fps. If you want to capture the image at 30fps, set the JPEG quality to 12 or more.
• If you want to direct write the image to the eMMC, set the 'Direct Write Mode' to checked. The image is directly written to the eMMC without the frame buffer. The writing speed is decreased. The default is unchecked.
• Changed the Allocate Unit Size to 32KB for the eMMC writing speed at the v0.3.3.

Update v0.3.1

• Added the movie shooting function. The movie is created by combining the images taken at the specified cycle. Capture Frames At Once (sec): The number of frames to capture at once. The default is 0, which means that the image is captured one by one. If you set it to 1, the image is captured during the 1 sec. If you set image resolution to VGA, the image is captured by 27 fps. HD is 15 - 27 fps. Full HD is 6 fps.
• Capture images are stored by appending the file. If you set to "Over Write Save", the image is overwritten.
• Added the Open AI model 'GPT-4o mini' selection. The default is 'GPT-4o'. You can select the model from GPT-4o, GPT-4 Turbo, GPT-4o mini.
• If touch the button during 3 secs on the back, the camera starts the movie shooting. The movie shooting is stopped when the button is touched again.
• Improved the writing speed of the eMMC.
• Change the Report Interval: The default is 3600 sec. The report interval is changed to 3600 sec. The status report is sent to LINE when the specified cycle is reached.
• Camera interface is changed. Power Down, Reset pin is removed. In previous version, the power down pin and reset pin are assigned by dumy GPIO pin. The 'esp-camera-rs' driver must be updated to the latest version.
• Fixed the bug that captured images are broken when the camera is in the DeepSleep mode. v0.3.1 is the stable version.
• Capturing clock is changed to 5MHz or 25MHz. The default is 5MHz when one image captured. When the movie is captured, the clock is changed to 25MHz.
• Changed the number of the frame buffer is 2. v0.3.0 set the number of the frame buffer to 3. But the eMMC writing speed is not enough to write the image to the eMMC. Since the eMMC writing speed is improved, the number of the frame buffer is changed to 2.
• Changed allocate_unit_size to 4096 for the eMMC writing speed at the v0.3.1.
• Fixed the bug that captured images are deleted when the camera is in the DeepSleep mode. v0.3.1 is fixed the bug.

Update v0.2.1

• Updated the UI for the configuration screen.
• Added the function to send a notification as status to LINE when the specified cycle is reached.
• Added the function to check the preview image on the web.
• Added images are uploaded to Cloudflare Images for private storage with signed URL.
• Added images are automatically deleted from the Cloudflare Images after 1 days.
• Added images can be downloaded as a compressed image from this camera on the web. (PC browser only)
• Added the function to check the remaining battery level on the web.
• Added the function to check the WiFi signal level on the web.
• Added the function to limit the time-lapse shooting time.
• Added the function to shoot when the camera touches the button during 3 secs on the back.

How it works

The TIME LEAP CAM operates by capturing images at pre-set intervals or specific times. These images are stored on the internal 64GB eMMC storage. During each capture, the camera performs AutoFocus to ensure clear images. The captured images are saved in one of the 12 available tracks, which can be configured as needed.

In monitoring mode, the captured images are sent to OpenAI's GPT-4o or GPT-4-Turbo along with a prompt (e.g., "If there is somebody, add 'NOTICE' in the reply and reason. If there is nobody, just reply 'NONE'."). The AI analyzes the image content based on the prompt. If the response contains "NOTICE, " it indicates that the image meets the specified conditions. If the response is "NONE, " it does not.

When "NOTICE" is detected, the image is uploaded to CloudFlare Images to generate a URL for the image. This URL, along with a notification message, is then sent to the LINE Messaging API. The designated LINE account receives the notification along with the image.

Notification conditions can be set using natural language, and the system supports multiple languages as long as they are supported by GPT-4o.

The camera is powered by a 3.7V LiPo 1100mAh battery, which is charged using a Qi wireless charger. Simply placing the camera on the Qi charger initiates the charging process, and charging stops automatically once the battery is full.

Configuration and image review can be done via a smartphone or PC browser over WiFi. Users can set parameters such as capture intervals, capture times, and resolution. Accessing the camera's IP address (e.g., http://192.168.2.100) opens the settings interface. The IP address is assigned by the DHCP server of the connected WiFi network and can be checked in the camera's startup console after flashing the firmware. Currently, the IP address, WiFi SSID, password, OpenAI API key, and CloudFlare access ID and API key must be set in the source code's cfg.toml file.

When the camera is in sleep mode, touching the back of the camera activates the web server, allowing access to the settings interface. If there is no interaction for 5 minutes, the camera automatically returns to sleep mode.

If capturing image intervals are over 90 seconds, the camera will be in deep sleep after capturing an image. If the duration is less than 60 seconds, the camera won't enter sleep mode. This is because the ESP32 Deep Sleep function runs an initialization sequence upon waking up, which takes about 30 seconds to complete.

The camera synchronizes its time using NTP when connected to WiFi and maintains the time using an RTC.

Design a Schematic

This is Qi Reciever schematic. I used BQ51013BRHLR for the control IC. AC1 and AC2 are connected to a receiver coil WR483265-15F5-G. If you connect the USB to a Host PC, this camera can charge to the battery from USB power.

ESP32-S3-WROOM, eMMC, and Camera Interface.

To reduce consumption power, eMMC and Camera power are controlled by ESP32-S3. By this control, power consumption is less than 1.5 mW. Pull-up registers for the eMMC line are 100 Kohms. It will suppress power consumption, too.

A camera interface has 24 pins connector. Pin 23, and 24 are used for the autofocus camera motor power. However, these pins are different from other OV5640 module products. I usedthe Seeed OV5640 Camera for ESP32S3 Sense. Other OV5640 modules sometimes are that pin23 is VDD, and pin24 is GND. It will not work.

PCB Layout

I used 4 layer Custom PCB.

I ordered PCBWay to manufacture my board. Click the "Add Gerber File" button, and you can upload the Gerber file.

When ordering the PCB, the parameters can be almost default values, but "Min hole size" should be 0.25mm, and "Immersion gold (ENIG)" should be selected for "Surface finish". The reason for this is that the minimum Via Hole is 0.25mm, and if the eMMC pad is not "Immersion gold (ENIG)", the solder on the pad will be slightly raised and cannot be soldered properly without misalignment.

After the order, I received boards about 1 week later.

After assembly, all components are installed.

You can order PCB and Assembly service on the PCBWay. This link page.

Install Software

The software for this camera is written in Rust. The compilation steps for Ubuntu 22.04 are detailed in the GitHub README. Please follow the steps below to install the software:

1. System Update and Package Installation

Update your system and install the necessary packages using:

sudo apt update && sudo apt -y install git python3 python3-pip gcc build-essential curl pkg-config libudev-dev libtinfo5 clang libclang-dev llvm-dev udev libssl-dev python3.10-venv

2. Rust Installation

Install Rust programming language and Cargo package manager:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

After installation, activate Rust by sourcing the environment:

. "$HOME/.cargo/env"

3. Additional Tools Installation

Install the following Rust tools:

• ldproxy
• espup
• cargo-espflash

Use the following commands:

cargo install ldproxy
cargo install espup
cargo install cargo-espflash

2024-08-12, cargo-espflash can not be compiled.

If you have an error, use the following command.

cargo install cargo-binstall
cargo binstall cargo-espflash

4. ESP Environment Setup

Run the following command to install and update the Espressif Rust ecosystem:

espup install
espup update

Set up environment variables:

. ./export-esp.sh

5. Udev Rules Configuration

Configure udev rules for device permissions:

sudo sh -c 'echo "SUBSYSTEMS==\"usb\", ATTRS{idVendor}==\"303a\", ATTRS{idProduct}==\"1001\", MODE=\"0666\"" > /etc/udev/rules.d/99-esp32.rules'
sudo udevadm control --reload-rules
sudo udevadm trigger

6. Clone Repository

Clone the TimeLeapCam repository:

git clone https://github.com/hnz1102/timeleapcam.git
cd timeleapcam/code/

7. Setting WiFi SSID, Password, etc.

Change the following configuration file.

cfg.toml

[timeleapcam]
wifi_ssid = "<Your AP SSID>"  # Set your AP SSID
wifi_psk = "<Your AP Password>" # Set your AP Password
http_server = "" # NOT USED
resolution = "8"
track_id = "0"
timezone_offset = "9"
auto_capture = "false"
idle_in_sleep_time = "300"
duration = "0"
api_key = "<API KEY for openAI>" # Set your OpenAI API Key
model = "gpt-4o"
query_openai = "false"
query_prompt = "If is there somebody, Add in the reply as 'NOTICE' and reason. If is nobody, Just reply is 'NONE'."
post_account = "<Your LINE Account ID>" # Set your LINE Account ID
post_access_token = "<Access Token for LINE Message>" # Set your LINE Access Token
storage_account = "<Your Cloudflare Account ID>" # Set your Cloudflare Account ID
storage_access_token = "Your Cloudflare Access Token>" # Set your Cloudflare Access Token
post_message_trigger = "NOTICE"

8. Build and Flash

Build the project and flash it to your device:

cargo build --release
cargo espflash flash --release --monitor

9. Run the Camera

After flashing, the camera will boot up. The console will display startup messages. The first boot may take a few minutes to format the eMMC. Once completed, an IP address will be displayed:

I (6520) esp_netif_handlers: sta ip: 192.168.2.191, mask: 255.255.255.0, gw: 192.168.2.1

Open a web browser and navigate to the displayed IP address (e.g., `http://192.168.2.191`). The configuration interface will allow you to set the capture interval, capture time, resolution, and other settings.

Ensure the camera is functioning correctly by checking the console output for messages indicating successful initialization:

I (9550) camera: Detected OV5640 camera
I (9550) camera: Camera PID=0x5640 VER=0x00 MIDL=0x00 MIDH=0x00
I (10250) cam_hal: buffer_size: 16384, half_buffer_size: 1024, node_buffer_size: 1024, node_cnt: 16, total_cnt: 960
I (10250) cam_hal: Allocating 983040 Byte frame buffer in PSRAM
I (10250) cam_hal: cam config ok
I (10270) ov5640: Set PLL: bypass: 0, multiplier: 200, sys_div: 4, pre_div: 2, root_2x: 0, pclk_root_div: 2, pclk_manual: 1, pclk_div: 4
I (10270) ov5640: Calculated XVCLK: 5000000 Hz, REFIN: 2500000 Hz, VCO: 500000000 Hz, PLL_CLK: 50000000 Hz, SYSCLK: 12500000 Hz, PCLK: 3125000 Hz
I (10310) timeleapcam::capture: Capturing Frame Thread Start...
I (10310) timeleapcam::monitoring: Query thread started
I (10310) timeleapcam::autofocus: Autofocus PID: 0x5640
I (10320) timeleapcam: Resolution changed: 21 -> 8
I (15270) timeleapcam::autofocus: Waiting for autofocus to start...
I (15270) timeleapcam::autofocus: Autofocus initialized
I (15290) timeleapcam::autofocus: AutoFocus Mode Enabled
I (15290) timeleapcam::autofocus: AutoFocus Triggered
I (25310) timeleapcam::autofocus: AutoFocus Timeout
I (25320) ov5640: Set PLL: bypass: 0, multiplier: 180, sys_div: 4, pre_div: 2, root_2x: 0, pclk_root_div: 2, pclk_manual: 1, pclk_div: 4
I (25320) ov5640: Calculated XVCLK: 5000000 Hz, REFIN: 2500000 Hz, VCO: 450000000 Hz, PLL_CLK: 45000000 Hz, SYSCLK: 11250000 Hz, PCLK: 2812500 Hz
I (25360) timeleapcam::autofocus: AutoFocus Mode Enabled
I (25360) timeleapcam::autofocus: AutoFocus Triggered
I (29490) timeleapcam::autofocus: AutoFocus Done

By following these steps, you will have successfully installed and configured the software for your Time Leap Cam.

Container Design

This camera is housed in a case created using a 3D printer. The case is designed with a certain level of water resistance to accommodate outdoor use, although it is not waterproof against submersion. At the base of the case, a coil is installed to receive power from a Qi wireless charger, and the battery is placed above this coil. The PCB is then mounted above the battery, with the camera portion protruding from the front.

A lid is attached to the base case to secure the internal PCB and camera components. Inside the case, there is a USB port for software writing and a screw that functions as a touch switch. A cover case is then placed over this assembly. The cover case includes a round acrylic plate to protect the camera lens while allowing it to be visible.

This case was created using a 3D printer. The STL data for the case can be downloaded from the following link. GitHub

Update Container

I created a base for the camera because I wanted to adjust the shooting angle. By embedding magnets inside the camera and the base, the camera's angle can be changed by attaching the magnets to each other. The outer cap has round protrusions to fix the angle, allowing it to be locked in place with a clicking sound. When charging the camera, the axis is connected by magnets, so the main body can be detached.

Using this Camera

Currently, I am trying out specific use cases and improving my software. I will update this article with any new useful cases I discover. following use cases are suggested by OpenAI.

• Elderly Monitoring and Emergency Notification

The camera is installed in the home of an elderly person to monitor their well-being. If the camera detects that the person has fallen or is in an abnormal state, it sends a notification to a designated caregiver or family member via LINE.

• Home Security Surveillance

The camera is used to monitor the home for security purposes. It can detect unusual activities such as an open window, a person entering the premises, or any other specified conditions.

• Pet Monitoring

The camera monitors pets at home, ensuring they are safe and have enough food and water. It can notify the owner if the pet's food bowl is empty or if the pet is in distress.

• Plant Health Monitoring

The camera is used to monitor the health of plants in a garden or indoor setting. It can detect if the plants are wilting or if they need watering.

• Weather Monitoring

The camera captures images of the sky to monitor weather conditions. It can notify the user if it detects signs of rain or other weather changes.

• Construction Site Monitoring

The camera is used on construction sites to monitor progress and ensure safety. It can detect if workers are wearing safety gear or if there are any safety hazards.

• Wildlife Observation

The camera is placed in natural habitats to observe wildlife behavior over long periods. It can notify researchers if specific animals are detected or if there are changes in the environment.

• Time-Lapse Photography for Projects

The camera is used to create time-lapse videos of long-term projects such as building construction, plant growth, or art creation.

• Remote Property Monitoring

The camera monitors remote properties such as vacation homes or cabins. It can detect and notify the owner of any unauthorized access or environmental changes.

Conclusion

In conclusion, the "TIME LEAP CAM" is an auto-focus camera designed for long-interval time-lapse photography and extended monitoring. It leverages the capabilities of OpenAI's GPT-4o/GPT-4o mini to analyze captured images and send notifications via LINE when specific conditions are met, such as detecting a person falling or a window being open. The camera operates on a low-power ESP32S3 SoC, supports WiFi communication, and uses Qi wireless charging. It features deep sleep mode for power conservation, a 64GB eMMC for storage, and a robust design suitable for outdoor use. The camera's settings can be easily configured through a web interface accessible via WiFi. This device addresses the need for automated monitoring without constant human supervision, making it a valuable tool for various applications, including elder care and environmental monitoring.

Finally, I would like to thank PCBWay for their support of the custom PCB board.

LICENSE

This source code is licensed under MIT. Other Hardware Schematic documents are licensed under CC-BY-SA V4.0.

This project uses the following libraries and resources:

https://github.com/jlocash/esp-camera-rs

https://github.com/0015/ESP32-OV5640-AF

These libraries are licensed under each library's respective licenses.