Shape Interaction

Design Process:

My initial goal is to have the Kinect working with Processing and obtaining some type of feedback from the user and its interaction with falling shapes.

11/21/2019

As a starting point, my objective was to be able to connect the Kinect with the computer and obtain some type of reading through Processing.

Throughout the research process I came across several codes and tutorials from Daniel Shiffman to program the Kinect to work with Processing. I began by testing the connection with the Kinect 1414 V1 for Xbox 360. Which, according to the code, it should work both with Kinect v1 and v2. However, after multiple attempts, it was not possible to make Processing read the Kinect v1, obtaining the following error:

After giving up on this method, I decided to do one last try with the Kinect v2 for Xbox One. According to many people, this version seemed to be better with the user detection and the compatibility with Processing.

From the beginning the software and driver installation was much more easy. I did struggle a bit with connecting it with Processing, but in the end it all worked out. Here's an example of the running code with depth detection:

Following is the code used for Kinect detection in Processing:

import org.openkinect.freenect.*;
import org.openkinect.freenect2.*;
import org.openkinect.processing.*;
import org.openkinect.tests.*;
Kinect2 kinect2;

void setup() {
size(512, 424, P3D);
kinect2 = new Kinect2(this);
kinect2.initDepth();
kinect2.initDevice();
}

void draw() {
background(0);
PImage img = kinect2.getDepthImage();

//Particle system based on the brightness detected
//int skip = 20;
//for (int x = 0; x < img.width; x++) {
//  for(int y = 0; y < img.height; y++) {
//    int index = x + y * img.width;
//    float b = brightness(img.pixels[index]);
//    float z = map(b, 0, 255, 250, -250);
//    fill(255-b);
//    pushMatrix();
//    translate(x, y, z);
//    rect(0,0,skip,skip);
//    popMatrix();
//  }
//}

image(img, 0, 0);
}

An issue I'm currently having is with the commented out code. It seems there is a lag with the image that is being retrieved. Not sure if this issue is due to my computer or other factors. I will determine this by testing on a different device.

In addition to connecting the Kinect, I started working on the creation of the shapes in Processing. Keeping in mind that I'm completely new to processing and Java, I need to learn this from the ground up.

I managed to create basic shapes and move them from the top to the bottom of the screen. I tried expanding on this by creating multiple shapes at the same time, but wasn't able to figure out how to move each of them individually. I found a great example, the PolygonPShapeOOP, within the Processing library which I will try to modify to my needs. For now here's an example of the falling shapes and the code used:

//Rectangle Y value
int y = 0;
//Circle X and Y values
int c = 0;
int r = 0;
//Triangle speed and each corner Y value
int m = 3;
int x1 = 150;
int x2 = 0;
int x3 = 150;

void setup(){
frameRate(2000);
size (1920, 1080);
}

void draw(){
background(0);
rect(100, y, 100,100);
y = y + 2;
ellipse(c, y, 100, 100);
c = c + 3;
r = r + 3;
triangle(850, x1, 950, x2, 1050, x3);
x1 = x1 + m;
x2 = x2 + m;
x3 = x3 + m;
}

Moving forwards my next step is to retrieve some type of data from the movements of the user. This data will be the one used to manipulate the shapes. In addition I will create the random falling shapes, make a crash effect, and hopefully apply a specific property to be manipulated in each.

12/13/2019

Till now I have made a lot of progress with the shape creation and interaction in Processing. I created multiple shapes that fall at random times, at random speed, and with random sizes. In addition to this, the user can interact with them through the mouse coordinates and each of the shapes has a different action that is performed when touched.

//Define the class for the shapes
Square[] manySquares = new Square[5];
Circle[] manyCircles = new Circle[5];
Triangle[] manyTriangles = new Triangle[5];
Diamond[] manyDiamonds = new Diamond[5];

//Load kinect libraries
import org.openkinect.freenect.*;
import org.openkinect.freenect2.*;
import org.openkinect.processing.*;
import org.openkinect.tests.*;
Kinect2 kinect2;

void setup(){
//size(512, 424); //size that is detectable for Kinect. Hopefully Kinect will be mapped to screen size: kinect.width = map(kinect.width,0,512,0,2736); & kinect.height = map(kinect.height, 0,424, 0,1824);

size(2736,1824);
for(int i = 0; i <manySquares.length; i ++){
manySquares[i] = new Square();
}
for(int i = 0; i <manyCircles.length; i ++){
manyCircles[i] = new Circle();
}
for(int i = 0; i <manyTriangles.length; i ++){
manyTriangles[i] = new Triangle();
}
for(int i = 0; i <manyDiamonds.length; i ++){
manyDiamonds[i] = new Diamond();
}
kinect2 = new Kinect2(this);
kinect2.initDepth();
kinect2.initDevice();
}

void draw(){
background(0);
PImage img = kinect2.getDepthImage();
image(img, 0, 0);
squ();
cir();
tri();
dia();
//Make shapes fall at different times
//if (frameCount > 1000) {
//  cir();
//}
//if (frameCount > 2000) {
//  tri();
//}
//if (frameCount > 3000) {
//  dia();
//}
//print(img);
}

void squ(){
//display many squares
for(int i=0; i < manySquares.length; i++){
manySquares[i].display();
manySquares[i].fall();
manySquares[i].move();
}
}

void cir(){
for(int i=0; i < manyCircles.length; i++){
manyCircles[i].display();
manyCircles[i].fall();
manyCircles[i].move();
}
}

void tri(){
for(int i=0; i < manyTriangles.length; i++){
manyTriangles[i].display();
manyTriangles[i].fall();
manyTriangles[i].move();
}
}

void dia(){
for(int i=0; i < manyDiamonds.length; i++){
manyDiamonds[i].display();
manyDiamonds[i].move();
manyDiamonds[i].fall();
}
}
//Future add ons:
//make speed increase after time
// make shape production start at 1 and increase over time

I created each shape class in a different tab to keep the code more organized.

class Square{
int x = int(random(width));
int y = int(random(-800,-300));
int size = int (random(100, 200));
PShape square;
int speed = int(random(2,6));
boolean right = false;
boolean left = false;

void display(){
fill(0, 204, 204);
noStroke();
square = createShape(RECT,x, y, size, size);
shape(square); // This makes the shape actually display. Based on the PShape variable name.
}

void fall(){
y = y+speed;
if(y > height + size){
restart();
}
}

void move(){
if (!right && mouseX > x && mouseX <= x+(size/2) && mouseY > y && mouseY < y+size) {
right = true;
left = false;
}
else if (!left && mouseX > x+(size/2) && mouseX <= x+size && mouseY > y && mouseY < y+size) {
left = true;
right = false;
}
if (right == true){
x = x+speed;
y = y-speed; // sort of works to keep it horizontal
}
else if (left == true) {
x = x-speed;
y = y-speed;
}
if (right == true || left == true){
if (y > height + size || x > width + size || x < 0 - size){
restart();
}
}
}

void restart(){
y = int(random(-200,-100));
x = int(random(width));
right = left = false;
}
}


class Circle{
int x = int(random(width));
int y = int(random(-800,-300));
int size = int(random(100, 200));
int r = size/2;
PShape circle;
int speed = int(random(2,6));
boolean se = false;
boolean sw = false;
boolean ne = false;
boolean nw = false;

void display(){
fill(185, 224, 9);
noStroke();
circle = createShape(ELLIPSE,x, y, size, size);
shape(circle); // This makes the shape actually display. Based on the PShape variable name.
}

void fall(){
y = y +speed;
if(y > height + (size*2)){
restart();}
}

void move(){
//Circle moves in diagonals
if(!se && mouseX > x-r && mouseX < x && mouseY > y-r && mouseY < y){
se = true;
sw = ne = nw = false;
}
else if(!sw && mouseX > x && mouseX < x+r && mouseY > y-r && mouseY < y){
sw = true;
se = ne = nw = false;
}
else if(!ne && mouseX > x-r && mouseX < x && mouseY > y && mouseY < y+r){
ne = true;
se = sw = nw = false;
}
else if(!nw && mouseX > x && mouseX < x+r && mouseY > y && mouseY < y+r){
nw = true;
se = sw = ne = false;
}
if (se == true){
x = x+speed;
y = y+speed;
}
else if (sw == true){
x = x-speed;
y = y+speed;
}
else if (ne == true){
x = x+speed;
y = y-(speed*2);
}
else if (nw == true){
x = x-speed;
y = y-(speed*2);
}
if ( se == true || sw == true || ne == true || nw == true){
if (y > height + (size*2) || x >= width + r || x <= x - size || y < -(size*2)){
restart();
}
}
}

void restart(){
y = int(random(-200,-100));
x = int(random(width - size));
se = sw = ne = nw = false;
}
}


class Triangle{
int x = int(random(width));
int y = int(random(-800,-300));
int size = int(random(100, 200));
int r = size/2;
int x1 = x - r;
int y1 = y + r;
int x2 = x;
int y2 = y - r;
int x3 = x + r;
int y3 = y + r;
PShape triangle;
int speed = int(random(2,6));
boolean touched = false;
boolean untouched = false;
//Smaller triangles to detect if mouse is inside triangle or not
float area;
float area1;
float area2;
float area3;
//float angle = 0; //for rotation speed

void display(){
fill(102,0,153);
noStroke();
triangle = createShape(TRIANGLE, x1, y1, x2, y2, x3, y3);
shape(triangle);
}

void fall(){
if (y2 > height + size){
restart();
}
y1 = y1 + speed;
y2 = y2 + speed;
y3 = y3 + speed;
}

void move(){
// detect when the triangle is touched. Calculate area of triangle and area of 2 vertices and MouseX/Y. If the last are equal to area then  it's touched.
float px = mouseX;
float py = mouseY;
float area = abs( (x2-x1)*(y3-y1) - (x3-x1)*(y2-y1) );
float area1 = abs( (x1-px)*(y2-py) - (x2-px)*(y1-py) );
float area2 = abs( (x2-px)*(y3-py) - (x3-px)*(y2-py) );
float area3 = abs( (x3-px)*(y1-py) - (x1-px)*(y3-py) );
if (!touched && area1 + area2 + area3 == area) {
touched = true;
untouched = false;
}
else if(!untouched && area1 + area2 + area3 == area){
//if touched a second time
touched = false;
untouched = true;
}
if (touched == true){
//triangle rotates when touched. Not working.
//triangle.rotateX(angle);
//angle = angle + .1;
y1 = y1 - (speed*5);
y2 = y2 - (speed*5);
y3 = y3 - (speed*5);
}
else if (untouched == true){
//triangle rotates in different direction when touched again. Not working.
//triangle.rotateX(angle);
//angle = angle - .1;
y1 = y1 + (speed*5);
y2 = y2 + (speed*5);
y3 = y3 + (speed*5);
}
if(touched == true && y1 < y - size || untouched && y2 > height + size){
restart();
}
}

void restart(){
y = int(random(-200,-100));
x = int(random(width - size));
x1 = x - r;
y1 = y + r;
x2 = x;
y2 = y - r;
x3 = x + r;
y3 = y + r;
touched = untouched = false;
}
}


class Diamond{
int x = int(random(width));
int y = int(random(-800,-300));
int size = int(random(100, 200));
int r = size/2;
int x1 = x - r;
int y1 = y;
int x2 = x;
int y2 = y - r;
int x3 = x + r;
int y3 = y;
int x4 = x;
int y4 = y + r;
PShape diamond;
int speed = int(random(3,8));
//4 triangles to detect when touching inside
float area;
float area1;
float area2;
float area3;
float area4;
boolean touched = false;
boolean untouched = false;
boolean notouch = true;

void display(){
fill(255, 153, 0);
noStroke();
diamond = createShape(QUAD, x1, y1, x2, y2, x3, y3, x4, y4);
shape(diamond);
}

void fall(){
if (y2 > height + size){
restart();
}
y1 = y1 + speed;
y2 = y2 + speed;
y3 = y3 + speed;
y4 = y4 + speed;
}

void move(){
// detect when the diamond is touched. Calculate area of the diamond and area of 2 vertices and MouseX/Y. If the last are equal to area then it's touched.
float px = mouseX;
float py = mouseY;
float area = abs( ((x2-x1)*(y3-y1) - (x3-x1)*(y2-y1) + (x3-x1)*(y4-y1) - (x4-x1)*(y3-y1)) );
float area1 = abs( (x1-px)*(y2-py) - (x2-px)*(y1-py) );
float area2 = abs( (x2-px)*(y3-py) - (x3-px)*(y2-py) );
float area3 = abs( (x3-px)*(y4-py) - (x4-px)*(y3-py) );
float area4 = abs( (x4-px)*(y1-py) - (x1-px)*(y4-py) );
if (!touched && area1 + area2 + area3 + area4 == area) {
touched = true;
untouched = false;
}
else if(touched && area1 + area2 + area3 + area4 == area){
//if touched a second time
touched = false;
untouched = true;
}
else if(area1 + area2 + area3 + area4 != area){
notouch = true;
}
if (notouch && touched == true){
x1 = x-(r/4);
x3 = x+(r/4);
notouch = false;
}
else if (notouch && untouched == true){
x1 = x - r;
x3 = x+r;
notouch = false;
}
if(touched == true && y2 > height + size || untouched == true && y2 > height + size){
restart();
}
}

void restart(){
y = int(random(-200,-100));
x = int(random(width - size));
x1 = x - r;
y1 = y;
x2 = x;
y2 = y - r;
x3 = x + r;
y3 = y;
x4 = x;
y4 = y + r;
touched = untouched = false;
}
}

I plan on expanding this code to make the shapes fall in increasing numbers. Starting with one shape ans slowly adding more, this way the users can have time to understand their way to interact with them.

As for the Kinect, after doing many tests in processing, I came to the conclusion that it's not possible to obtain skeletal data from this specific library. I tried working with a different library, but for some reason the Kinect doesn't turn on when using that code. Due to this, I will be revising my original idea in terms of how the interaction with the user will be included. I will need to base the interaction on depth data and possibly locations on the screen. This will also allow the interaction of multiple users, something I will keep in mind to possibly include some action depending on the amount of users present at the same time.

My next step will be to determine the extent of the Kinect in terms of user interaction, and define how this interaction will be in order to revise my idea within the actual abilities of the library.

03/05/2020

Finally after a lot of testing I was able to obtain the Kinect raw depth data to interact with the falling shapes. I also included an obstacle and point system which are still in coding process.

Although this interaction is essentially what I was looking for, the size of the projection is too small. Since the Kinect only detects a resolution of 512 x 424, the resulting image is of the same size. I tried presenting the projection on full screen and expanding the resulting Kinect image using the image() function, but the issue here was that the data detection and interaction was still happening within the same 512 x 424, instead of full width and height.

import org.openkinect.processing.*;
// Kinect Library object
Kinect2 kinect2;
Square[] manySquares = new Square[3];
float minThresh = 0;
float maxThresh = 850;
int point = 0;
boolean in = false;
boolean out = true;
PImage img;
int d;
int offset;
int sx = int(random(512));
int sy = int(random(-100,-200));
int ssize = int (random(30, 80));
PShape square;
int sspeed = int(random(1,3));
boolean sright = false;
boolean sleft = false;
int obsColor = (255);

void setup() {
size(512, 424);
background(0);
kinect2 = new Kinect2(this);
kinect2.initDepth();
kinect2.initDevice();
//makes a blank image
img = createImage(kinect2.depthWidth, kinect2.depthHeight, RGB);
for(int i = 0; i <manySquares.length; i ++){
manySquares[i] = new Square();
} }

void draw() {
//we will set images on the pixel based on the raw depth data
img.loadPixels();
int[] depth = kinect2.getRawDepth();
for (int x = 0; x < kinect2.depthWidth; x++) {
for (int y = 0; y < kinect2.depthHeight; y++) {
offset = x + y * kinect2.depthWidth;
d = depth[offset];
if (d > minThresh && d < maxThresh) {
img.pixels[offset] = color(255,0,150);
if (!sright && x > sx && x <= sx+(ssize/2) && y > sy && y < sy+ssize ) {
sright = true;
sleft = false;
}
else if (!sleft && x > sx+(ssize/2) && x <= sx+ssize && y > sy && y < sy+ssize) {
sleft = true;
sright = false;
}
if(sleft && sx <= 22 && sy < height-100) {
sx=22;
sright = true;
sleft = false;
}
}
else {
img.pixels[offset] = color(0);
} } }
points();
img.updatePixels();
image(img, 0, 0);
//Place all assets of the screen after the kinect pixel update
fill(255);
textSize(32);
text(point, width/4, 45);
obstacles();
squ();
}

void squ(){
//display many squares
for(int i=0; i < manySquares.length; i++){
manySquares[i].display();
manySquares[i].fall();
manySquares[i].move();
} }

void obstacles(){
fill(obsColor);
noStroke();
rect(0,0,20,height-100);
stroke(obsColor);
strokeWeight(5);
line(width/2,0,width/2,500);
}

void points(){
if(sx + ssize > -ssize && sx < (width+ssize) && sy + ssize > -ssize && sy < (height+ssize)){
in = true;
out = false;
}
else{
in = false;
out = true;
}
if (in && sx < 0) {
point++;
} }

class Square{
void display(){
fill(0, 204, 204);
noStroke();
square = createShape(RECT,sx, sy, ssize, ssize);
shape(square); // This makes the shape actually display. Based on the PShape variable name.
}

void fall(){
sy = sy+sspeed;
if(sy > height + ssize){
restart();
} }

void move(){
if (!sright && mouseX > sx && mouseX <= sx+(ssize/2) && mouseY > sy && mouseY < sy+ssize) {
sright = true;
sleft = false;
}
else if (!sleft && mouseX > sx+(ssize/2) && mouseX <= sx+ssize && mouseY > sy && mouseY < sy+ssize) {
sleft = true;
sright = false;
}
if (sright == true){
sx = sx+sspeed;
sy = sy-sspeed; // sort of works to keep it horizontal
}
else if (sleft == true) {
sx = sx-sspeed;
sy = sy-sspeed;
}
if (sright == true || sleft == true){
if (sy > height + ssize || sx > width + ssize || sx < 0 - ssize){
restart();
} } }

void restart(){
sy = int(random(-200,-100));
sx = int(random(width));
sright = sleft = false;
ssize = int (random(30, 80));
sspeed = int(random(1,3));
} }

//camera information based on the Kinect v2 hardware
static class CameraParams {
static float cx = 254.878f;
static float cy = 205.395f;
static float fx = 365.456f;
static float fy = 365.456f;
static float k1 = 0.0905474;
static float k2 = -0.26819;
static float k3 = 0.0950862;
static float p1 = 0.0;
static float p2 = 0.0;
}

After testing many different and unsuccessful options in Processing to expand the size and interaction of the Kinect, I decided to try using the projector as a way of zooming into the image and presenting it as full size. I found that changing the computer's screen resolution and adding zoom to the projector resulted in a somewhat usable projection. However, the quality is not as great as I wish, but it is an option that would potentially work.

Although this is an option, I couldn't stop there. After trying many more options and different codes, and after seeking for help with the code, I was able to make the interaction in full screen by mapping out the small 512 x 424 resolution to a full screen size. This option maps each pixel and spreads them out on the projected size, forming the image out of many small dots (the pixels). I will keep trying to find a way for the image to be a solid color. However, since the projection will be over a large background, this pixel based image shouldn't be an issue.

PImage img;
PImage img2;

void setup() {
size(2736,1824,P2D); //fullscreen size - Need to test with fullscreen(P2D);
img = createImage(kinect2.depthWidth, kinect2.depthHeight, RGB);
img2 = createImage(width, height, RGB);
for(int i = 0; i <manySquares.length; i ++){
manySquares[i] = new Square();
} }

void draw() {
img.loadPixels();
img2.loadPixels();
int[] depth = kinect2.getRawDepth();

for (int x = 0; x < kinect2.depthWidth; x++) {
for (int y = 0; y < kinect2.depthHeight; y++) {
int offsetImg1 = x + y * kinect2.depthWidth ;
int xo = int(map(x, 0, kinect2.depthWidth, 0, width));
int yo = int(map(y, 0, kinect2.depthHeight, 0, height));
int offsetImg2 = xo + yo * width;
int d = depth[offsetImg1];
if (d > minThresh && d < maxThresh) {
img2.pixels[offsetImg2] = color(250,0,250);
} else {
img2.pixels[offsetImg2] = color(0,0,0);
} } }

//NEW CODE TO DETECT IN FULL SCREEN 
for(int x = 0; x < width; x++) {
for( int y = 0; y < height; y++) {
color c = get(x,y);
// if c is pink, collision is detected
if(c == color(250, 0, 250)) {
if (!sright && x > sx && x <= sx+(ssize/2) && y > sy && y < sy+ssize ) {
sright = true;
sleft = false;
}
else if (!sleft && x > sx+(ssize/2) && x <= sx+ssize && y > sy && y < sy+ssize) {
sleft = true;
sright = false;
}
if(sleft && sx <= 22 && sy < height-100) {
sx=22;
sright = true;
sleft = false;
} } } } }

Having solved the issue of the projection size, I now need to find a way for the rendering to be quick. Then I will continue with the coding of the shapes and define what functions the interface will have for the users to interact with.