Integrating Google Gemini with Home Assistant and RPi Pico W

We will see how to control the Raspberry Pi Pico W using voice commands. Also, we talk with Google Gemini through Node-RED dashboard and Home Assistant operating system in voice and text. Also, the Raspberry Pi Pico W microcontroller sends the temperature and humidity to the Home Assistant via MQTT, and we can see the temperature and humidity graphically in the Node-RED dashboard.

see full video on my youtube channel:

We start our work from the Node-RED dashboard. As you can see in the image below, in one section, we can have a voice conversation with Google Gemini by pressing the microphone button. Also, in the dashboard, we can hear Gemini answer by voice. In the Gemini Responses section, the text of Gemini answer can be seen. If you don't like to have a voice conversation with Gemini, you can easily enter your text through the question box, so that Gemini will answer you. On the right side of the dashboard, you will also see another microphone button. When you press this button, you can control the Raspberry Pi Pico by voice commands. For example, when I say "light on", a lamp turns on. You can also control other devices with different voice commands. Finally, we can see the temperature and humidity graphically. I have also placed a conditional function block in the Node-RED diagram. For example, if the temperature in the output of this function falls below a certain limit, Gemini will offer us a hot drink according to the air temperature.

Next, you can see the block diagram of Node-RED logic. We can easily implement a complex program without messing with complex programming and network concepts. In this block diagram, using the MQTT in and MQTT out blocks, we can exchange data with the Raspberry Pi Pico W microcontroller and a Python script that is running the Gemini program.

You can use the Node-RED block diagram through the following script.

[{"id":"2e86cc79543dd949","type":"ui_microphone","z":"ab42f0c2b344442b","name":"controller","group":"673ac7a1e346be05","order":1,"width":0,"height":0,"maxLength":5,"maxRecogLength":5,"timeslice":0,"press":"press","mode":"recog","interimResults":false,"x":80,"y":280,"wires":[["6d2432cc9a571c09"]]},{"id":"673ac7a1e346be05","type":"ui_group","name":"controller","tab":"4c2a2d8b4284e446","order":3,"disp":true,"width":"6","collapse":false,"className":""},{"id":"4c2a2d8b4284e446","type":"ui_tab","name":"Egoo","icon":"dashboard","disabled":false,"hidden":false}]

Since we have connected to Home Assistant through the browser without an SSL certificate. The browser does not allow the Node-RED dashboard to use the device's microphone due to security issues. In order to be able to use the microphone of our own device (mobile phone, computer), we must change the browser settings and give this permission to Home Assistant to use the device's microphone. Changing the security settings of browsers are different and even changing the settings of mobile browsers is much more difficult than Windows.

We said that Node-RED exchanges information with a Python script via MQTT. You can see this python script below.

import google.generativeai as genai
import paho.mqtt.publish as publish
import random
import paho.mqtt.client as mqtt
from time import sleep
# MQTT parameters
MQTT_SERVER = "192.168.1.232" #mqtt broker ip
MQTT_PATH = "test" #topic
port = 1883
'''
broker = '192.168.1.232'
port = 1883
topic = "test"
'''
final_msg = []
client_id = f'python-mqtt-{random.randint(0, 1000)}'
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc, properties=None):
client.username_pw_set('mqtt', 'sharifi77mm98')
print("Connected with result code "+str(rc))
client.subscribe("question")
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
#print(msg.payload.decode())
message = msg.payload.decode()
final_msg.append(message)
print(message)
# Create an MQTT client and attach our routines to it.
# Specify the new callback API version
client = mqtt.Client(client_id=client_id, callback_api_version=mqtt.CallbackAPIVersion.VERSION2)
client.on_connect = on_connect
client.on_message = on_message
client.connect(MQTT_SERVER, port, 60)
# Process network traffic and dispatch callbacks.
client.loop_start()
genai.configure(api_key="AIzaSyDSFt21TMaIhJOahjg6YxRA2DStFrG7sAY")
model = genai.GenerativeModel('gemini-1.0-pro-latest')
while True:
if final_msg:
nodered_message = final_msg.pop(0)
#print(nodered_message)
response = model.generate_content(str(nodered_message))
print(response.text)
publish.single("response", response.text, hostname=MQTT_SERVER, auth={'username':"mqtt", 'password':"sharifi77mm98"})
sleep(1)
else:
continue

When we write text in the question box or speak through the Talk to Gemini microphone, a text data is sent to the Python script above via the MQTT protocol. This Python script gives the input data to Gemini by means of Google API. Finally, Gemini's response is sent to Node-RED via MQTT.

In the last section we have the Raspberry Pi Pico W microcontroller. We have two functions, one function sends temperature and humidity sensor information to Home Assistant via MQTT protocol, and the other function receives various control commands from Home Assistant and Node-RED via MQTT protocol. Using the Multicore Programming method, we execute these two functions on the microcontroller. This program is written in Micro Python language.

import network
from time import sleep
from umqtt.simple import MQTTClient
import ujson
from machine import Pin, ADC
import dht
import _thread
dht_sensor = dht.DHT11(Pin(15))
analog_value = ADC(Pin(26))
light = Pin(14,Pin.OUT)
def air_params():
#sleep(2)
dht_sensor.measure()
temp = dht_sensor.temperature()
hum = dht_sensor.humidity()
#CO_value = analog_value.read_u16()
return temp, hum
# Network credentials
WIFI_SSID = 'Mrsh77'
WIFI_PASSWORD = '1m77n2299215r77#'
# MQTT credentials
MQTT_BROKER = '192.168.1.232'
MQTT_PORT = 1883
MQTT_TOPIC = 'pico'
msg_from_node_red = []
# Connect to WiFi
def connect_wifi(ssid, password):
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
if not wlan.isconnected():
print('Connecting to network...')
wlan.connect(ssid, password)
while not wlan.isconnected():
pass
print('Network config:', wlan.ifconfig())
# Callback function to handle messages
def on_message(topic, msg):
#print('Received message:', msg)
msg_from_node_red.append(msg)
# Setup MQTT client and connect to the broker
def setup_mqtt():
client = MQTTClient(client_id='pico_w_client', server=MQTT_BROKER, port=MQTT_PORT, user="mqtt", password="sharifi77mm98")
client.set_callback(on_message)
client.connect()
client.subscribe(MQTT_TOPIC)
print('Connected to %s, subscribed to %s topic.' % (MQTT_BROKER, MQTT_TOPIC))
return client
connect_wifi(WIFI_SSID, WIFI_PASSWORD)
client = setup_mqtt()
def send_params():
while True:
temperature, humidity = air_params()
sensor_data = {
"temperature": temperature,
"humidity": humidity
}
json_data = ujson.dumps(sensor_data)
client.publish("pico_in",json_data)
sleep(1)
def core1_task():
try:
while True:
client.check_msg()
if msg_from_node_red:
final_msg = msg_from_node_red.pop(0)
print(final_msg)
if final_msg == b'light on':
light.value(1)
elif final_msg == b'light off':
light.value(0)
sleep(1)
except KeyboardInterrupt:
print('Disconnected from MQTT Broker.')
finally:
client.disconnect()
_thread.start_new_thread(core1_task, ())
send_params()

The electronic circuit of the microcontroller is very simple and you can even see how to connect the sensor and relay from the code.