Team Wacky Weirdos: Creating Art from Data: Lucas Atayde, user1051395, Ranferi Avilez

Published April 23, 2019

Wacky Weirdos: Creating Art with Data

Our project turns environmental data (temperature, humidity, sound) into beautiful works of mathematical art.

IntermediateShowcase (no instructions)8,915

Things used in this project

Hardware components

Texas Instruments EK-TM4C123GXL TM4C Tiva LaunchPad

Seeed Studio Grove Starter Kit for LaunchPad

Software apps and online services

The Processing Foundation Processing

Texas Instruments Energia

Apple macOS

Story

Inspiration:

Our project's inspiration came from a YouTube video (https://youtu.be/fDSIRXmnVvk ). The video shows how "chaos equations", or in actuality, parametric equations, can be used to create art. The parametric equations are affected by variable t that represents time, which can be incremented, or reset, by a simple code.

Peripherals :

The video gave us the idea to use the peripherals from the SEEED Grove Starter Kit for Launch Pad to change the variables of the equations that would draw art for us. We chose peripherals that would be able to record information about the environment around us: the sound sensor, the temperature and humidity sensor, as well as the light sensor. Unfortunately, we realized that our light sensor was very poor quality / had factory defects, so we opted to use the rotary angle sensor instead. Unfortunately, this sensor responds to user input instead of environmental factors, but it still resulted in a really cool end product!

With the peripheral instruments chosen, we began to work on recording the data from our surrounding environment. The code on our Tiva Microcontroller reads in the information about the sound, temperature, humidity, and rotary angle sensor. The board then compresses these inputs down to a value between 0 and 255. We compress the data down to a point in which we can represent each sensor as two hexadecimal digits. With the four sensors, we send a total of 8 hexadecimal digits over the serial port to the processing code. This allows serial communication to happen at a much faster rate than if we were to send over the raw data. Additionally, it standardizes the data to it's easier to work with in Processing.

After a few hours of coding up a way to draw parametric equations in Processing, we realized that parametric equations are kinda boring. We wanted another way to create cool mathematical art that was more complex and dynamic. We eventually decided upon vector fields, as they can become extremely complex and often replicate patterns found in nature. We wanted to procedurally generate these vector field functions, but we realized that algorithms for generating new, random functions are more complicated than one might expect. Additionally, incorporating sensor data into randomly generated functions doesn't make much sense. For example, it's easy to think of a way in which to generate random polynomial functions with our data, but what about terms like sin, cos, or even sigmoid? Due to the sheer number of different equations one can make, we ultimately decided upon a piecewise function instead. As a result, our code technically has a finite number of different pieces of art, or vector fields, it can generate. Luckily, we included enough variation into the piecewise function that it generally produces art not seen in the past.

The code itself it surprisingly simple. After brainstorming for a few hours, we found the best way to make art in a vector field is by having particles trace the effects of the fields on their motion. Each particle on the screen is a unique object. Each of these particles, has a location vector as well as a velocity vector. Each tick (similar to a frame), the velocity vector for the individual particle is calculated, then added to its location vector. The resulting movement traces the path along a vector field. The following equations exactly model how the vector fields change based on sensory information as well as location on the screen:

1 / 2 • Vector Field Parametric Equation

For the particle classes themselves, our code borrows from the following example code provided by processing. However, we heavily modified the class as well as the code to apply to our specific use. More can be learned about Processing here:https://processing.org/tutorials/pvector/

The Team Abuses a Macbook:

The next step to building it was scaling it up. One particle is cool and all, but it doesn't really reveal much about the vector field or produce very cool looking art. So in the code itself, we have a giant array of these particle objects, each with its own location and velocity. The computer the code ran on, a 2015 Macbook Pro with an i7 Core, could surprisingly track about 10, 000 particles at once without threading. At first, we found that our code was extremely slow, making the art laggy. In search of a way to solve this lag, we introduced threading.

In layman's terms, threading is essentially the process of splitting up tasks in code. On a lower level, it essentially tells your processor to run another task simultaneously as your main body of code. Processing provides a really easy way to use threading within the code. Namely, it provided some helper functions that will take care of most of the work of threading in the background. This was convenient for us because manually introducing threading into code can be a serious challenge. Through these functions, we can tell the computer to essentially work on calculations in parallel, instead of telling the computer to read in data, calculate the new velocities, and update the screen all at once.

We used this concept of threading to greatly increase the number of particles we could render. At its peak, I was able to get my macbook to render approximately 300, 000 particles on the screen at once, albeit at 10 frames per second. I found that a "good" number of particles for my computer to render while maintaining a solid frame rate is around 45, 000. We used threading in the application to divide the task of generating new velocities up. Inside an array of 40, 000 particles, we create four different threads to run calculations on chunks of 10, 000 particles each. Each of these functions are ran simultaneously, each action on different parts of an array in memory. While the different threads are working on calculating the new positions, the main section of code can simply focus on drawing the particles on the screen. While this technically allows for some particles to be "behind" due to the threads not being in sync with the main drawing thread, it's not noticeable at all. I imagine if one were to rewrite this code to utilize a graphics card this asynchronous rendering would create some issues. For the sake of the project as well as our own understanding, it doesn't create issues.

Overall Flow:

The following structure can be used to understand how we designed the code to operate. Unfortunately, we couldn't get serial interrupts to work in the code. As a result, the processing code simply reads in data from the serial buffer in the main thread when the spacebar is pressed.

Processing works just like Energia! It has two functions: a setup function and a loop function. The setup does a few things like set up serial communication and the size of the window, but the fun happens inside of the loop.

Inside of the main loop, the code starts by checking the last saved value from the sound sensor to determine if particles or lines will be rendered, based on a certain threshold. Next, it dispatches 4 different threads to recalculate the position of 1/4 of the particles each. While those threads are running, we loop through the same array of particles and draw them on the screen. This goes back to my note earlier about the weakness of our implementation. Because the threads are modifying the particles in the same memory locations as we're rendering them on the screen, one can imagine situations in which particles get out of sync. For example, the particle at index 0 will have a newer position value then the particle at index 39, 999. We avoided this problem by making sure that the threads act evenly across the particles on the screen. Instead of saying, "let's split the particles into chunks of 10, 000 based on the top left, top right, bottom left, and bottom right of the screen, " we evenly select particles across the screen to be recalculated at once. This prevents any weird lag or rolling shutter effects. As a consequence it might just create a small amount of imprecision / noise in the vector field. Seeing as our main goal with using threads is to speed up the frame rate as well as increase the number of particles on the screen, this effect is pretty much impossible to see thanks to the persistence of vision.

After the code has looped through each of the particles to draw on the screen, it then checks to see if they spacebar has been pressed. This ideally would use an interrupt instead, but we couldn't get it to work with the specific combination of Processing + Threading + Energia + Tiva. As a result, we simply have to use the "pinging" method on the spacebar, a concept we learned about in class.

After we have "pinged" the spacebar, processing automatically runs the function again.

Examples:

Here are some of the cooler pictures we rendered!

1 / 5

We had a lot of fun on this project! Send us an email at lsa4@rice.edu if you have any questions!

Lucas Atayde, Joshua Verne, Ranferi Aviles.

Code

// IMPORTS NECESSARY FOR SERIAL COMMUNICATION 
import org.quark.jasmine.*;
import processing.serial.*;

float e = 2.718281828459045;
float counter = 0;

// Number inside of array determines how many particles rendered on the screen
// Lower / Raise this number depending on your computer. 40,000 is about the max on my computer before
// my computer starts to choke. A gaming laptop can probably handle quite a bit more.
Mover[] movers = new Mover[40000];
Serial myPort; // The serial port object necessary for reading in values.

//Global Integers track values from each of the corresponding sensors.
// By default, they're all set to half. Each value was normalized on the board to be between 0-255, 0 being no signal, 255 meaning maximum signal.
int temp = 127;
int humi = 127;
int sound = 127;
int toggle = 127;


//Runs once at startup, just like Energia
void setup () {
  //P2D allows faster rendering on some computers. Remove the line inside the parenthesis if it doesn't work / isn't compatible with your computer
  fullScreen(P2D);
  //Prints out list of serial ports. THIS WILL VARY DEPENDING ON COMPUTER. 
  //Not sure if this will work on windows, but play around with the correct serial port until serial communication works
  println(Serial.list());
  //Set Serial.list[NUMBER] according to the corresct serial port attached
  myPort = new Serial(this, Serial.list()[3], 9600);
  //Looks for new line character
  myPort.bufferUntil('\n');
  //Sets background color to black. 
  background(0); 
  //Initialized mover objects (particles) according to number allocated at top of code
  for (int i = 0; i < movers.length; i++) {
    movers[i] = new Mover(); 
  }
  
}


//This giant code block loops pretty much as quickly as your computer can handle it.
void draw(){
  
  //This determines if "particles" or "lines" are drawn on the screen. Uses the value input from the mic.
  // I found that using 28 was a good balance for the mic we used. Essentially, if the sound reading is
  //greater than 28, it will redraw the background every frame. If below, it never removes the last frame creating a path
    if (sound > 28){
      background(0);
      //Stroke weight determines radius of points drawn
      strokeWeight(2);
    } else {
      strokeWeight(1);
    }  
    
    
    
    //This code should hopefully work on your computer. Creates new threads to update the points, that way the processor putting 
    // drawing the image isn't bogged down waiting for new calculations
    thread("update_points_1");
    thread("update_points_2");
    thread("update_points_3");
    thread("update_points_4");
    
    
    //Loops through each particle / mover to display them on the screen.
    for (int i = 0; i < movers.length; i++) {
      movers[i].display();
    }
    
    
    //Draws the information about the sensor readings in the top left corner of the screen.
    textSize(32);
    text("TEMP: " + temp + " HUMI: " + humi + " SOUND: " + sound + " ROTARY: " + toggle, 10, 30); 
    fill(256,256,256);
    
    
    //Checks if spacebar has been presses
    if (keyPressed){
      if (key == ' ')
          //Checks the Serial Buffer to see if anything is avaliable to read 
          if ( myPort.available() > 0) { 
            //Read in the value from the serial port as a string. This value will be 8 characters long, every two representing a number
            // between 0 and 255 in HEX
            String inString = myPort.readString();
            //Keep this delay here. Dealing with serial ports is pretty annoying and rushing it will cause things to crash :(
            delay(150);
            //Clears the Serial Buffer.
            myPort.clear();
            if (inString.length() > 0) {
                 //
                 // First Pair of Charactes = Hex Value for temperature reading
                 // Second Pair of Characters = Hex Value for humidity reading
                 // Third Pair of Characters = Hex Value recorded from mic
                 // Fourth Pair = Hex Value for Rotary Angle Sensor
                 inString = trim(inString);
                 //Decodes each of the Hex values
                 temp = Integer.decode("0x" + inString.substring(0,2));
                 humi = Integer.decode("0x" + inString.substring(2,4));
                 sound = Integer.decode("0x" + inString.substring(4,6));
                 toggle = Integer.decode("0x" + inString.substring(6,8));
               
                 // Wipes out the background to draw the new vector field
                 background(0);
                 //Resets the position of all the particles Allows him to interact with vector field.
                 for (int i = 0; i < movers.length; i++) {
                    movers[i].reset();
                   }
             }
          }   
      }
    }
  }
  

//Starts Thread 1 for calculating the new position for 1/4th of the points.
public void update_points_1(){
  for (int i = 0; i < movers.length/4; i++) {
      movers[i].update(humi,temp,sound,toggle);
      movers[i].checkEdges();
    }
}

//Starts Thread 2 for calculating the new position for 1/4th of the points.
public void update_points_2(){
  for (int i = movers.length/4; i < movers.length/2; i++) {
      movers[i].update(humi,temp,sound,toggle);
      movers[i].checkEdges();
    }
}

//Starts Thread 3 for calculating the new position for 1/4th of the points.
public void update_points_3(){
  for (int i = movers.length/2; i < 3*(movers.length)/4; i++) {
      movers[i].update(humi,temp,sound,toggle);
      movers[i].checkEdges();
    }
}

//Starts Thread 4 for calculating the new position for 1/4th of the points.
public void update_points_4(){
  for (int i = 3*(movers.length)/4; i < movers.length; i++) {
      movers[i].update(humi,temp,sound,toggle);
      movers[i].checkEdges();
    }
}


class Mover{ 
  
  //Vector Objects for tracking location and velocity.
  PVector location;
  PVector velocity;
  float fx; // Derivative in i-hat direction
  float fy; // Derivative in j-hat direction
  
  //Saves values from sensors
  float mTemp;
  float mSound;
  float mHumi;
  float mToggle;

  //Creates mover objects. Set random location on the screen. Sets velocity to zero
  Mover() {
    location = new PVector(random(width),random(height));
    velocity = new PVector(0,0);
  }
  
  //Resets the location of the mover and sets the velocity to zero
  void reset(){
    location = new PVector(random(width),random(height));
    velocity = new PVector(0,0);
  }
  
  //Function called by the threads to update the particles / movers
  void update(int humi, int temp, int sound, int toggle) {
    //Updates values in object. I copy the values because I found some weird bugs when using threading.
    mTemp = (float) temp;
    mSound = (float) sound;
    mToggle = (float) toggle;
    mHumi = (float) humi;
    
    //I map the vector's location to a value between -20 and 20. This makes typing in vector field equations easier.
    float x = (map(location.x, 0, width, -200, 200))/10.0;
    float y = (map(location.y, 0, height, -100, 100))/10.0;
    
    //Monster sized piece wise function. Mathematically described in the project writeup. Basically uses the different values from
    // the sensors to decide the properties of the vector field.
    if (mToggle >= 0) {
      fx = sin(x + y);
      fy = cos(x + y);
    }
    if (mToggle > 100){
      fx = sin(x - y);
      fy = sin(x + y);
    }
    if (mToggle > 150){
      fx = sigmoid(y) + cos(x)*log(abs(y));
      fy = sigmoid(x/10);
    }
    if (mToggle > 170){
       fx = log(abs(y));
       fy = log(abs(x));
    }
    if (mToggle > 180){
       fx = log(abs(y)) - log(abs(x));
       fy = log(abs(y)) + log(abs(x));
    }
    if (mToggle > 210){
       fx = log(abs(y)) + log(abs(x));
       fy = log(abs(y))*log(abs(x));
    }
    
    
    //Humidity modifiers
    if (mHumi % 10 == 0) {
      fy = cos(x + y);
    }
    if (mHumi % 10 == 1){
      fy = sin(x + y);
    }
    if (mHumi % 10 == 2){
      fy = -5*fy;
    }
    
    //Temperature modifiers
    if (mTemp % 10 == 0) {
      fx = sigmoid(sin(x))*10;
    }
    if (mTemp % 10 == 1){
      fx = sigmoid(fy);
    }
    if (mTemp % 10 == 2){
      fx = -5*fx;
    }
    
    // Sound modifiers
    if (mSound % 7 == 0){
      fx = fy;
    } else if (mSound % 7 == 1){
      fx = fy*-fx;
    }
    
    
    //Updates velocity + location
    velocity = new PVector(fx,fy);
    location.add(velocity);
  }

  void display() {
    //Sets color depending on color and location
    float r = ((mTemp % 15) /5) * 256;
    float g = (location.x / width) * 256;
    float b = (location.y / height) * 256;
    stroke(r,g,b);
    //draws the point on the screen
    point(location.x,location.y);
  }
  void checkEdges() {
    //When particles go off the screen, they're respawned.
    if (location.x > width) {
      reset();
    } else if (location.x < 0) {
      reset();
    }

    if (location.y > height) {
      reset();
    }  else if (location.y < 0) {
      reset();
    }
  }
}


//We covered this function in 182. I use it to create some pretty cool warping effects.
float sigmoid(float val){
    float sig = 1;
    sig = sig / (1 + pow(e,val));
    return sig;
}

#include "TM1637.h"
#include "DHT.h"
#include "Ultrasonic.h"


// Different pins used to receive values
#define TEMP_HUMI_PIN 23
#define SOUND_PIN 24
#define SPIN_PIN 25
#define SEND_PIN 17

// Variables that store un-compressed sensor data
double _sound = 0;
double _toggle = 0;
double _temp = 0;
double _humi = 0;
double _dist = 0;

//Sets up Temperature Humidity Sensor
DHT dht(TEMP_HUMI_PIN, DHT22);

void setup() {
  //Starts Serial communication with the computer at 9600 baud
  Serial.begin(9600);
  dht.begin();
}

void loop() {
  //The number commented next to each value correspond the range I found I could get out of our sensors.
  //The sensors aren't very high quality so often times they don't work super well.
  _temp = (double) dht.readTemperature(); //0-100
  _humi = (double) dht.readHumidity(); //0-100
  _sound = (double) analogRead(SOUND_PIN); //0 - 4095 (MAX I CAN GET IT TO IS AROUND 2200);
  _toggle = (double) analogRead(SPIN_PIN);

  //Converts each of the values to a number between 0 and 255
  int t_temp = (_temp / 100 ) * 255;
  int t_humi = (_humi / 100 ) * 255;
  int t_sound = min((_sound / 2000) * 255,2000);
  int t_dist = min((_dist / 200) * 255,200);
  int t_toggle = (_toggle / 4095 ) * 255;

  //After each number converted to be between 0 and 255, we bitshift them to fit into an 8 bit data type. We used an unsigned integer to store the values
  unsigned int transpose = (t_temp << 24) + (t_humi << 16) + (t_sound << 8) + (t_toggle);
  //Send the compressed value over the serial port in hex
  Serial.println(transpose, HEX);
  //Delay by 50 milliseconds.
  delay(50);
}

Wacky Weirdos: Creating Art with Data