Kindbot: Home Garden Automation Hub

See what's new at Remyx.ai

The Dawn of Kindbot

At the turn of 2018, we began developing a plant monitor to explore home grown cannabis in California. Cannabis flowers are valued at an average $320 oz, taking months to develop under ideal growing conditions.

Typically, home grow enthusiasts will set up small spaces or tents where they can hang HID/LED lights, fans for ventilation, AC for temperature control and irrigation pumps for hydroponic feeding.

Growers often rely on timers to coordinate the scheduling for all of these appliances and excess heat tends to be a challenge to maintaining the stability that helps flowers to thrive.

In addition to this equipment, growing great produce requires some domain knowledge in biology & plant science. Forums and pricey apps help to fill the knowledge gap but advice is seldom on-demand and frequently ill-informed either by the bias of how an issue is framed to the advisor or by the experiences of those offering advice.

Environmental controller technology is geared toward commercial applications in pricing and assumptions on the user experience.

In Kindbot, we developed a system to maintain ideal growing conditions which leverages deep learning in environmental control and plant health diagnostics.

To make Kindbot the most powerful and accessible environmental controller, we use a simple hardware design and promote computer vision with the picamera.

Ultimately, we have the ideal platform to support:

• maintaining stable, optimal environmental control
• remote access with photos and dashboard via mobile app
• rich data logging & smart notifications

Building Kindbot

We begin with the compact and flexible Raspberry Pi Zero microcontroller and explored a range of atmospheric sensors including each of the DHT11, Si7021, BME280 before settling on the latter.

At the bare minimum, we require temperature and humidity readings. From these values, we turn to the Arrhenius equations to calculate the Vapor-Pressure deficit (VPD). This statistic informs a frame of reference contextualizing temperature and humidity in terms of physical pressures driving photosynthesis in the plant.

Informed by this statistic, we can target different ranges to support the natural growth cycles optimally.

By logging our sensor data with sqlite3, we found some sporadic readings to be unreasonably anomalous. For a robust controller, we will need to be aware of this kind of error.

Since our it is our goal to maintain stable growing conditions, we employ a simple 'temperature continuity in time' assumption. Simply put, we ignore readings which deviate too much over a short time scale.

Once we can trust and log our readings, we can whip up a slackbot to report on current conditions, send SMS notifcations with twilio when conditions fall out of target range, or email reports to get an overview to inspect for any anomaly indicative of a problem.

While designing Kindbot, we've tested the YL-69 sensor to probe soil moisture as well as the TSL-2561 to measure light intensity, before making greater use of the camera to streamline the device profile.

In our efforts to control the most important environmental parameters, we even developed a sister device, the Budtender, to perform pH control and autodosing of nutrients.

While emphasizing a modern, consumer electronics aesthetic over traditional environmental controller equipment, we explored integration with smart home devices. We've dabbled with VUI running a flask-ask server with Alexa Smart Home Skills and flask-assistant Google Home Assistant Actions.

Ultimately, we aim for a very simple design and take our UI mobile in both iOS and android.

Kindbot Cool

In the end, we want more than logging and notifications, we want control. One simple idea would be to periodically check the latest temperature reading and apply the simple 'on-off' decision based on whether the temperature exceeds a predetermined threshold. This decision would inform whether our AC is running over the next cycle before we reevaluate in N minutes. However, this often leads to undesirable oscillations in temperature.

The standard approach is to use PID control. Here, is a fine overview on heuristics used to tune your PID models. With some trial and error, we can achieve reasonable stability around the set point of 74 F without too much overshooting when the lights kick on each morning.

We also explored applications of deep learning by reframing our temperature control problem as a game within a reinforcement learning paradigm.

For example, imagine putting an agent into an environment where allowable actions are to turn the AC on or off for the next N-minute cycle. At the beginning of each cycle, we evaluate our state, which may as well include temperature, humidity, and any other important, recent environmental statistics.

Then if we are on target, to within a small tolerance plus or minus the set point, our agent is rewarded. On the other hand, if the temperature is outside the acceptable tolerance, our agent is penalized in a way proportional to the degree of this difference.

An agent like the above can learn a policy function in an online manner to specialize environmental control with the REINFORCE algorithm aka Policy Gradients.

Kindbot helps to reduce energy use, control excessive heat, and boost metabolism by manipulating the temperature and its changes.

Kindbot, Feed Me!

Controlling irrigation is often much simpler and schedule-based. A little research informs decisions in watering frequency based on the irrigation designs whether hydroponic, aeroponic, or soil. To Kindbot, it always boils down to knowing which smart plug to control and how long/often to run it.

We make use of the pyHS100 repo to control appliances like irrigation pumps, lights and fans. You can even query energy utilization with the TP-Link HS110 smart plug.

In another woody, annual plant: poinsettas, research shows better root development for cuttings using dynamic irrigation control with irrigation frequency proportional to current VPD measurements.

We can transform a range of VPD values into a range of misting frequencies so that we irrigate more frequently to provide additional cooling and use natural hydraulic forces on the plant to drive nutrient uptake in aeroponic or drip irrigation systems.

Kindbot Cannavision

Since people proudly share photographs of their cannabis grows, we had a wealth of open source data to explore computer-vision based techniques to extract valuable information from images of our grow space.

We began with what we could gather with simple google image searches, taking advantage of a very handy firefox plugin. We can expand our result set by refining our search queries, trying other search engines, and reverse image searching good prototype images.

In this way, we gather several thousand photos and annotate each to identify important plant structures using labelimg. This helps us to explore yield or harvest time estimation by identifying bud structures, or notify the grower about yellowing vegetation which could signify a nutrient/pH issue requiring a human touch, or even to recognize a limp, droopy leaf indicating low water pressure so that Kindbot can trigger an irrigation event.

Just as we log temperature readings, we also log computer vision model results and this helps us to average out noise between images for more a more robust alerting system.

Another simple idea is to threshold images in HSV colorspace so that we can measure the proportion of green pixels in the picamera field of view. The following code block can be used to quantify the development of vegetative mass with the leaf area index.

img = cv2.imread('YOUR_IMG_FILE.png', 1)
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
mask = cv2.inRange(hsv, (36, 0, 0), (70, 255,255))
imask = mask>0
green = np.zeros_like(img, np.uint8)
green[imask] = img[imask]
green = np.where(green > 0, 1, 0)
lai = np.mean(green)

Selecting a couple photos taken by Kindbot, we threshold to isolate sufficiently green pixels.

Calculating the proportion of green pixels in these images, which we assume is vegetation, we have the Leaf Area Index.

The following plot shows how our calculation of leaf area changes over time for the grow pictured above. We observed explosive growth under the consistent conditions Kindbot helped to maintain.

Kindbot applies this information to:

• offer feedback in camera positioning
• track weekly relative increase in growth
• change light schedules to trigger fruiting after sufficient veg
• notify user of light outage

Kindbot Generally

Lighting conditions in growing environments can differ considerably from one setup to another. Just look at the image search results for 'LED grow light':

Our calculation of the Leaf Area Index required thresholding images in HSV color space. However, unnatural light temperatures can greatly impact this measurement.

We explored GANs to learn image transformations to perform color correction:

Because our initial object detection models relied on many thousands of sample objects exemplifying many common grow room configurations, this model remained quite robust to differences in color temperature.

Nonetheless, we did not stop here in gathering data. Starting with ImageNet and its links for cannabis images, we crawled the web gathering many more images. Further, we developed a specialized classifier called Buddy to diagnose issues due to visual indicators of environmental, pest, and nutrient/water stressors.

Tools like the python request and lxml libraries help here. Getting comfortable with xpath will make it easy. Scrapy takes things up a notch and if you are really trying to build up your dataset, check out this post on incorporating the SOCK5 proxy behind polipo.

The Kindbot Shell

Kindbot began with a computer and a collection of environmental sensors before we aggressively eliminated those for which computer vision offers proxy information.

As we compactify the device we tested several designs and materials. Likewise, we explored different finishing techniques for Kindbot using wood stains, polycrylic, sanding and polishing PLA materials and acetone vapor baths for ABS.

Initial designs made use of a slick ball and socket swiveling attachment.

Most recently, we have ditched the ball and socket joint for a slimmer profile while using the standard camera mounting hardware so users can take advantage of readily available accessories.

Kindbot packaging ideas...

Kindbot Beyond

Every grower should at least monitor temperature and humidity. Many will opt to introduce a webcam to their indoor garden for remote monitoring and security. Often gardeners rely on mechanical timers to program their irrigation system.

Kindbot rolls all of this important functionality into a tiny device packing AI to extract valuable insights from the image feed to perform actions based on sound gardening science.

Computer vision applications have made their way into agtech, though early work relied on simple methods which can be fairly sensitive to subject's size and shape. Applying machine learning to more robustly identify plant stress phenotype is a relatively modern effort. Some research uses deep learning to identify plant structures with an encoder-decoder neural network architecture. Some use classical machine learning. Others get more data using a 3D laser scanner as the payload of an industrial robotic arm.

With an image classification model trained to identify dozens of plant toxicities, deficiencies, and pests, we built a service to diagnose plants with a simple image upload.

We believe the this will form the basis for smart irrigation systems which conserve water. It can have applications outside of cannabis cultivation such as food production in space.

We can see how the results transfer beyond cannabis cultivation with this garden tomato plant example.

Kindbot irrigation control can be improved through the introduction of an autodoser with pH control. This is the mission behind a related product which we call the Budtender.

Github repo

Acknowledgements

We would like to thank:

• UC Berkeley Moffit Library Makerspace for free 3D printing & training
• Berkeley Indoor Garden for insightful conversations
• Bitraf Makerspace Oslo, Norway for hospitality
• Growers Network for spotlighting us on this new technology

Update: 4/20/18 party at UC Berkeley's Memorial Glade, we handed out nearly 100 samples of organic, pesticide-free bud from our inaugural harvest to promote Kindbot.