Hands on SenseCap Watcher

In the last few months I have been developing an application that uses artificial intelligence and object recognition. I started my project using an AiThinker ESP32-Cam, then moved on to a Xiao Esp32-S3 Sense and finally, to the Grove Vision AI V2, which turned out to be the best choice of all.

However, the SenseCap Watcher has an excellent proposal, and incorporates exactly the same hardware as the Grove Vision AI V2, with an incredible user interface, ease of use and the possibility of integrating LLMs, image recognition, voice and integration with other sensors via GPIO bus.

The features are being implemented little by little by the Seeed Studio team, but for me this device has the potential to be the Arduino of artificial intelligence, facilitating the development of simple automation applications.

The product page on Kickstarter has all the necessary information and the folks at Seeed have produced excellent documentation that allows users to get started.

Unboxing

The product packaging has an excellent design and upcycles into a stand for the Watcher. In the package, a USB type C cable, a universal wheel base and a 1/4 female adapter set.

Task programming by LLM

For now, the device can be programmed using a specific cell phone application, called SenseCraft Mate App, which is available for Android and iPhones. All programming is done by assigning tasks using a simple chat interface. It is worth mentioning that the software is capable of understanding other languages, although I only tested it with my native language, which is Brazilian Portuguese....

As I mentioned before, the Seeed team created excellent documentation, detailing the entire initial configuration of the device, which is quite simple.

Connect USB cable to your computer

When you connect your Watcher via USB, Windows registers two new serial ports. In my case, COM23 and COM24. I decided to do an experiment and see what would appear on a serial terminal, because I am a Circuitpython fan, and I love the REPL! For this I used Putty and configured it to use the serial port at 115200 baud rate (I was lucky, the first COM23 port was already the one I needed to discover the command console). But the other one, COM24 is used to upgrade firmware using Xmodem protocol in the same way as Grove Vision AI V2.

Right away, the terminal displays a series of information about the device's memory. If you press enter, a prompt appears and you can type some commands that I list below:

help  [<string>]

Print the summary of all registered commands if no arguments are given, otherwise print summary of given command, where <string> is the name of command

wifi_sta  [-s <ssid>] [-p <password>]

WiFi is station mode, join specified soft-AP

• s <ssid> SSID of AP
• p <password> password of AP

ota  [-t <int>] [--url=<string>]

force ota, ignoring version check

• t, --ota_type=<int> 0: ai model, 1: himax, 2: esp32
• -url=<string> url for ai model, himax or esp32 firmware

taskflow  [-iej] [-f <string>]

Import taskflow by json string or SD file, eg:

taskflow -i -f "test.json".

Export taskflow to stdout or SD file, eg:

taskflow -e -f "test.json"

• i, --import import taskflow
• e, --export export taskflow
• f, --file=<string> File path, import or export taskflow json string by SD, eg: test.json
• j, --json import taskflow json string by stdin

factory_info

Get factory infomation

battery

Get battery percent

bsp  subcmd [subcmd args]

Bsp call bsp functions

bsp i2cdetect <0|1>

Scan the specified i2c bus

reboot

Reboot the device

factory_reset

Factory reset and reboot the device

New stuff added August 18, 2024!

record  [-t <int>] [-f <string>]

Record audio and save to SD.

-t, --time=<int> record time, s

-f, --file=<string> File path, Store PCM audio data in SD card

vi_ctrl  [-sec] [-z <int>]

Voice interaction ctrl (Probably will be used with Push to Talk function)

-s, --start start wakeup, and start record

-e, --end end record

-c, --stop stop voice interaction when analyzing or palying, Put it into idle.

-z, --exit=<int> 0: exit vi, 1:exit vi then run new taskflow

iperf  [-suVa] [-c <ip>] [-p <port>] [-l <length>] [-i <interval>] [-t <time>] [-b <bandwidth>]

Check conection performance. Similar to iperf linux command.

-c, --client=<ip> run in client mode, connecting to <host>

-s, --server run in server mode

-u, --udp use UDP rather than TCP

-V, --ipv6_domain use IPV6 address rather than IPV4

-p, --port=<port> server port to listen on/connect to

-l, --len=<length> Set read/write buffer size

-i, --interval=<interval> seconds between periodic bandwidth reports

-t, --time=<time> time in seconds to transmit for (default 10 secs)

-b, --bandwidth=<bandwidth> bandwidth to send at in Mbits/sec

-a, --abort abort running iperf

rgb  [-r <int>] [-g <int>] [-b <int>] [-m <int>] [-v <int>] [-t <int>]

Set the rgb LED value. eg: rgb -r 255 -g 0 -b 0 -m 3

-r, --red=<int> red value, 0~255

-g, --green=<int> green value, 0~255

-b, --blue=<int> blue value, 0~255

-m, --mode=<int> 1: breath, 2: blink, 3:solid, default 3

-v, --step_value=<int> RGB step value, default 3

-t, --step_time_ms=<int> RGB step time(ms), default 5

Taskflow

With an SD Card inserted and using the command taskflow -e -f "test.json", it is possible to output a json file that should be something similar to this one:

{
  "tlid": 1723668142381,
  "ctd": 1723668148345,
  "tn": "Notify and say 'face detected'",
  "task_flow": [
    {
      "id": 1303304955,
      "type": "ai camera",
      "type_id": 0,
      "index": 0,
      "vision": "0.0.1",
      "params": {
        "model_type": 0,
        "model": {
          "model_id": "60094",
          "version": "1.0.0",
          "arguments": {
            "size": 790.23,
            "url": "https://sensecraft-statics.oss-accelerate.aliyuncs.com/refer/model/1705457676206_ci8tA3_swift_yolo_1xb16_300e_coco_300_int8_sha1_2287b951101007d4cd1d09c3da68e53e6f23a071_vela.tflite",
            "icon": "https://sensecraft-statics.oss-accelerate.aliyuncs.com/refer/pic/1715062849589_HpZ46y_FaceDetection.png",
            "task": "detect",
            "createdAt": 1705457694,
            "updatedAt": 1715062857,
            "iou": 45,
            "conf": 50
          },
          "model_name": "Face Detection",
          "model_format": "tfLite",
          "ai_framework": "6",
          "author": "SenseCraft AI",
          "description": "The model is a Swift-YOLO model trained on the human face detection dataset.",
          "task": 1,
          "algorithm": "Object Dectect(TensorRT,SMALL,COCO)",
          "classes": [
            "face"
          ],
          "checksum": "4dbee281d6c32bdb3ba47b2d63907c1a"
        },
        "modes": 0,
        "conditions": [
          {
            "class": "face",
            "mode": 1,
            "type": 2,
            "num": 0
          }
        ],
        "conditions_combo": 0,
        "silent_period": {
          "time_period": {
            "repeat": [
              1,
              1,
              1,
              1,
              1,
              1,
              1
            ],
            "time_start": "00:00:00",
            "time_end": "23:59:59"
          },
          "silence_duration": 60
        },
        "output_type": 1,
        "shutter": 0
      },
      "wires": [
        [
          363272002
        ]
      ]
    },
    {
      "id": 363272002,
      "type": "image analyzer",
      "type_id": 3,
      "index": 1,
      "version": "0.0.1",
      "params": {
        "url": "",
        "header": "",
        "body": {
          "prompt": "Face Detected",
          "type": 1,
          "audio_txt": "Face detected"
        }
      },
      "wires": [
        [
          2118011860,
          762085127
        ]
      ]
    },
    {
      "id": 762085127,
      "type_id": 99,
      "type": "sensecraft alarm",
      "index": 2,
      "version": "0.0.1",
      "params": {
        "silence_duration": 10,
        "text": "Face detected"
      },
      "wires": []
    },
    {
      "id": 2118011860,
      "type_id": 5,
      "type": "local alarm",
      "index": 4,
      "version": "0.0.1",
      "params": {
        "sound": 1,
        "rgb": 1,
        "img": 1,
        "text": 1,
        "duration": 10
      },
      "wires": []
    }
  ],
  "type": 0
}

Json File Clues

For now we don't have access to the SenseCap Watcher API (only SenseCap API is available), but the Json file generated by a task gives us several interesting indications.

It is worth mentioning that you can configure it to output JSON via UART and use the processed information to control other devices, such as Arduino, Raspberry Pi, ESP32 and others. There is still a lack of clarity regarding the parameters indicated in the file, but as soon as the API is available, it will be possible to use this capacity to create even more complex tasks.

The Json file gives us some clues as to what information the system uses to download the models and my attention is drawn to the "wires" parameters that appear several times. Will there be any way to control external hardware devices via I2C, UART or GPIO? Probably yes. The version 1.1.3 indicates that there is a possibility to access an external temperature and humidity sensor in I2C bus (SGP30).

Seeed's approach to creating this device gave me some ideas: Will it be possible to write device drivers using just their task system? This would be of great help in compatibility of the device with the large network of sensors and peripherals already available for platforms such as Arduino and Raspberry Pi.

Boot Dump

And finally, a dump with the system boot messages, which appear after the reboot command is executed on the serial terminal.

I (15425688) wifi:state: run -> init (0)
I (15425689) wifi:pm stop, total sleep time: 12991594922 us / 15389024413 us
I (15425690) wifi:<ba-del>idx:0, tid:0
I (15425701) wifi:new:<6,0>, old:<6,0>, ap:<255,255>, sta:<6,0>, prof:1
I (15425703) app-wifi: wifi event: WIFI_EVENT_STA_DISCONNECTED
I (15425705) app-wifi: Other disconnection reason: 8
I (15425716) app-wifi: retry to connect to the AP
I (15425733) wifi:flush txq
I (15425733) wifi:stop sw txq
I (15425734) wifi:lmac stop hw txq

▒▒▒ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0xc (RTC_SW_CPU_RST),boot:0x9 (SPI_FAST_FLASH_BOOT)
Saved PC:0x40375ef8
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3820,len:0x1948
load:0x403c9700,len:0x4
load:0x403c9704,len:0xf64
load:0x403cc700,len:0x30c4
entry 0x403c995c
I (31) boot: ESP-IDF v5.2.1-1-g3a705cf126 2nd stage bootloader
I (31) boot: compile time Jul  1 2024 22:32:04
I (31) boot: Multicore bootloader
I (36) boot: chip revision: v0.2
I (39) qio_mode: Enabling QIO for flash chip WinBond
I (45) boot.esp32s3: Boot SPI Speed : 80MHz
I (50) boot.esp32s3: SPI Mode       : QIO
I (54) boot.esp32s3: SPI Flash Size : 32MB
I (59) boot: Enabling RNG early entropy source...
I (65) boot: Partition Table:
I (68) boot: ## Label            Usage          Type ST Offset   Length
I (76) boot:  0 nvsfactory       WiFi data        01 02 00009000 00032000
I (83) boot:  1 nvs              WiFi data        01 02 0003b000 000d2000
I (90) boot:  2 otadata          OTA data         01 00 0010d000 00002000
I (98) boot:  3 phy_init         RF data          01 01 0010f000 00001000
I (105) boot:  4 ota_0            OTA app          00 10 00110000 00c00000
I (113) boot:  5 ota_1            OTA app          00 11 00d10000 00c00000
I (121) boot:  6 model            Unknown data     01 82 01910000 00100000
I (128) boot:  7 storage          Unknown data     01 82 01a10000 005f0000
I (136) boot: End of partition table
I (140) esp_image: segment 0: paddr=00d10020 vaddr=3c170020 size=4e43f8h (5129208) map
I (934) esp_image: segment 1: paddr=011f4420 vaddr=3fca1c00 size=06574h ( 25972) load
I (938) esp_image: segment 2: paddr=011fa99c vaddr=40374000 size=0567ch ( 22140) load
I (944) esp_image: segment 3: paddr=01200020 vaddr=42000020 size=165578h (1463672) map
I (1172) esp_image: segment 4: paddr=013655a0 vaddr=4037967c size=18500h ( 99584) load
I (1191) esp_image: segment 5: paddr=0137daa8 vaddr=600fe010 size=00004h (     4) load
I (1204) boot: Loaded app from partition at offset 0xd10000
I (1204) boot: Disabling RNG early entropy source...
I (1216) cpu_start: Multicore app
I (1216) octal_psram: vendor id    : 0x0d (AP)
I (1216) octal_psram: dev id       : 0x02 (generation 3)
I (1220) octal_psram: density      : 0x03 (64 Mbit)
I (1225) octal_psram: good-die     : 0x01 (Pass)
I (1231) octal_psram: Latency      : 0x01 (Fixed)
I (1236) octal_psram: VCC          : 0x01 (3V)
I (1241) octal_psram: SRF          : 0x01 (Fast Refresh)
I (1247) octal_psram: BurstType    : 0x01 (Hybrid Wrap)
I (1253) octal_psram: BurstLen     : 0x01 (32 Byte)
I (1258) octal_psram: Readlatency  : 0x02 (10 cycles@Fixed)
I (1265) octal_psram: DriveStrength: 0x00 (1/1)
I (1271) MSPI Timing: PSRAM timing tuning index: 5
I (1275) esp_psram: Found 8MB PSRAM device
I (1280) esp_psram: Speed: 80MHz
I (1736) esp_psram: SPI SRAM memory test OK
I (1746) cpu_start: Pro cpu start user code
I (1746) cpu_start: cpu freq: 240000000 Hz
I (1746) cpu_start: Application information:
I (1749) cpu_start: Project name:     factory_firmware
I (1755) cpu_start: App version:      1.0.1
I (1760) cpu_start: Compile time:     Jul 15 2024 08:33:52
I (1766) cpu_start: ELF file SHA256:  97eb5a61f...
I (1771) cpu_start: ESP-IDF:          v5.2.1
I (1776) cpu_start: Min chip rev:     v0.0
I (1781) cpu_start: Max chip rev:     v0.99
I (1786) cpu_start: Chip rev:         v0.2
I (1791) heap_init: Initializing. RAM available for dynamic allocation:
I (1798) heap_init: At 3FCAB328 len 0003E3E8 (248 KiB): RAM
I (1804) heap_init: At 3FCE9710 len 00005724 (21 KiB): RAM
I (1810) heap_init: At 3FCF0000 len 00008000 (32 KiB): DRAM
I (1817) heap_init: At 600FE014 len 00001FD4 (7 KiB): RTCRAM
I (1823) esp_psram: Adding pool of 8171K of PSRAM memory to heap allocator
I (1831) spi_flash: detected chip: winbond
I (1835) spi_flash: flash io: qio
W (1840) i2c: This driver is an old driver, please migrate your application code to adapt `driver/i2c_master.h`
I (1850) sleep: Configure to isolate all GPIO pins in sleep state
I (1857) sleep: Enable automatic switching of GPIO sleep configuration
I (1864) coexist: coex firmware version: 77cd7f8
I (1870) coexist: coexist rom version e7ae62f
I (1875) main_task: Started on CPU0
I (1879) esp_psram: Reserving pool of 256K of internal memory for DMA/internal allocations
I (1888) main_task: Calling app_main()
I (1892) :
 _____                      _________    ____
/ ___/___  ____  ________  / ____/   |  / __ \
\__ \/ _ \/ __ \/ ___/ _ \/ /   / /| | / /_/ /
___/ /  __/ / / (__  )  __/ /___/ ___ |/ ____/
/____/\___/_/ /_/____/\___/\____/_/  |_/_/     WATCHER
--------------------------------------------------------
Version: 1.0.1 Jul 15 2024 08:35:31
--------------------------------------------------------
I (1933) MEM:     Biggest /  Minimum /     Free /    Total
DRAM : [  188416 /   286454 /   286410 /   580318]
PSRAM: [ 8257536 /  8365120 /  8365120 /  8367812]
DMA  : [  188416 /   278522 /   278522 /   572170]
I (2112) : # SN: ?????????????
I (2112) : # EUI: ?????????????
I (2113) : # CODE: ?????????????
I (2114) : # DEVICE_KEY: ?????????????
I (2120) : # BATCHID: ?????????????
I (2123) : # ACCESS_KEY: ?????????????
I (2129) : # AI_KEY: ?????????????
I (2133) : # DEV_CTL_KEY: ?????????????
I (2138) : # PLATFORM: 0
I (2600) BSP: Partition size: total: 5715521, used: 1759008
I (2601) BSP: Initialize IO I2C bus
I (2601) gpio: GPIO[2]| InputEn: 1| OutputEn: 0| OpenDrain: 0| Pullup: 1| Pulldown: 0| Intr:2
I (2710) BSP: IO expander initialized: ffd9
I (2712) LVGL: Starting LVGL task
I (2712) spd2010: LCD panel create success, version: 1.0.1
I (2871) BSP: Initialize knob input device
I (2871) gpio: GPIO[41]| InputEn: 1| OutputEn: 0| OpenDrain: 0| Pullup: 1| Pulldown: 0| Intr:3
I (2874) gpio: GPIO[42]| InputEn: 1| OutputEn: 0| OpenDrain: 0| Pullup: 1| Pulldown: 0| Intr:3
I (2883) Knob: Iot Knob Config Succeed, encoder A:41, encoder B:42, direction:0, Version: 0.1.4
I (2893) button: IoT Button Version: 3.2.3
I (2897) BSP: Initialize I2C bus
I (2902) BSP: Initialize touch panel
I (2907) SPD2010: Touch panel create success, version: 0.0.1
I (3063) gpio: GPIO[40]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 1| Pulldown: 0| Intr:0
I (3068) ES8311: Work in Slave mode
I (3079) I2S_IF: channel mode 0 bits:16/16 channel:2 mask:1
I (3079) I2S_IF: STD Mode 1 bits:16/16 channel:2 sample_rate:16000 mask:1
I (3090) Adev_Codec: Open codec device OK
I (3091) I2S_IF: channel mode 0 bits:16/16 channel:2 mask:2
I (3093) I2S_IF: STD Mode 0 bits:16/16 channel:2 sample_rate:16000 mask:2
I (3101) Adev_Codec: Open codec device OK
I (3105) gpio: GPIO[46]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 1| Pulldown: 0| Intr:0
I (3115) sscma_client.io.spi: spi max trans bytes: 8184
I (3425) deviceinfo: usage_guide_switch value loaded from NVS: 1
I (3426) deviceinfo: device info init early done, qrcode content: w1:??????????????????
I (3434) BSP: voltage: 4190mV
I (3437) BSP: voltage: 4190mV
I (3441) BSP: voltage: 4194mV
I (3444) BSP: voltage: 4190mV
I (3448) BSP: voltage: 4190mV
I (3452) BSP: voltage: 4190mV
I (3455) BSP: voltage: 4194mV
I (3459) BSP: voltage: 4190mV
I (3463) BSP: voltage: 4190mV
I (3466) BSP: voltage: 4190mV
I (3470) BSP: percentage: 100%
W (3612) sscma_client: request not found: INIT@STAT?
W (4637) PNG Load: No images found with primary prefix Custom_greeting, trying secondary prefix greeting
I (4913) PNG Load: Loaded greeting1.png into PSRAM
I (4967) PNG Load: Loaded greeting3.png into PSRAM
I (5720) PNG Load: Loaded greeting2.png into PSRAM
W (7073) PNG Load: No images found with primary prefix Custom_detecting, trying secondary prefix detecting
I (7674) PNG Load: Loaded detecting1.png into PSRAM
I (8547) PNG Load: Loaded detecting3.png into PSRAM
I (9499) PNG Load: Loaded detecting2.png into PSRAM
I (10782) PNG Load: Loaded detecting5.png into PSRAM
I (11954) MEM:     Biggest /  Minimum /     Free /    Total
DRAM : [  188416 /   175766 /   213634 /   580318]
PSRAM: [ 6422528 /  6461768 /  6462208 /  8367812]
DMA  : [  188416 /   175762 /   213890 /   572170]
I (12160) PNG Load: Loaded detecting4.png into PSRAM
W (14561) PNG Load: No images found with primary prefix Custom_detected, trying secondary prefix detected
I (16114) PNG Load: Loaded detected2.png into PSRAM
I (17194) PNG Load: Loaded detected1.png into PSRAM
W (18352) PNG Load: No images found with primary prefix Custom_speaking, trying secondary prefix speaking
I (19787) PNG Load: Loaded speaking1.png into PSRAM
I (20540) PNG Load: Loaded speaking3.png into PSRAM
I (20575) PNG Load: Loaded speaking2.png into PSRAM
W (21836) PNG Load: No images found with primary prefix Custom_listening, trying secondary prefix listening
I (21965) MEM:     Biggest /  Minimum /     Free /    Total
DRAM : [  188416 /   175766 /   220854 /   580318]
PSRAM: [ 6160384 /  6267180 /  6268272 /  8367812]
DMA  : [  188416 /   175762 /   220850 /   572170]
I (22108) PNG Load: Loaded listening2.png into PSRAM
I (22169) PNG Load: Loaded listening1.png into PSRAM

Boot Message Analysys

Boot messages give us some clues about available hardware and capabilities. The user interface uses the excellent LVGL library. The touch display is an Solomon Systems SPD2010 connected via I2C via GPIO40 pin. The I2S audio chipset is the ES8311 and is connected via the GPIO46 pin. The rotary encoder is connected to GPIO41 and GPIO42 pins.

Native Home Assistant Integration

One of the projects I'm working on requires integration with HomeAssistant, which I still haven't managed to configure on GroveVision AI V2, even following all the steps in Seeed's tutorial. With Watcher, integration was immediate. It seems to me that it is still at a very preliminary stage, because the indicators that appear on the HomeAssistant Dashboard have nothing to do with the camera, microphone or speaker. Instead, we have temperature, pressure and humidity indicators, probably inherited from other devices in the SenseCap family.

Full Source Code of Firmware

The source code of firmware: https://github.com/Seeed-Studio/SenseCAP-Watcher-Firmware

Final thoughts

My first impressions of the device are extremely positive! The list of features promised by Seeed is truly incredible and I hope that the partnerships with Ultralytics, HomeAssistant, Nvidia Tao, Node-Red really happens.

If I were you, I'd run to Kickstarter right away!