Super96s Clusters - Final Integration

----------------------------------------------------------------------------------------------------------------

Link

Schematic: https://www.avnet.com/opasdata/d120001/medias/docus/193/Ultra96-V2%20Rev1%20Schematic.pdf

Board files: https://github.com/Avnet/bdf

Kubernetes Lightweight (k3s): https://k3s.io/

----------------------------------------------------------------------------------------------------------------

Objective

The objective of this module is to make that final pass in integrating the whole system as a cluster. The last couple modules showed you and explored what an overlay was. You are probably wondering and asking the question - where is this cluster? This is where this module comes into play.

The goal is to connect: [4] Ultra96s - [1] PYNQ-Z1 - [1] Raspberry-pi 4. Showing this range, provides insight that you are not device dependant nor do you need to strictly use Xilinx devices. The Overlays explore PYNQ, the (k3s) explore 3rd party alternatives that can join a cluster that are not dependant on PYNQ. I'm trying to show you that having a cluster can be scalar and can unleash a realm of possibilty - from FPGAs to MCUs.

There are a couple of ways to start this adventure! The best way is to first locate your devices name (ip address), this will allow you to identify the following nodes below. Once you have the names, we will ssh into all devices and download a package that will help you quickly talk to your nodes. They will then share the same path directories, the files you will be running are different. You will see this in a second. It makes it easier to execute tasks in the command line that stretches to all the devices sharing the overlay.

Below are the functions you will see :

• Master
• Node1
• Node2
• Node3
• Node4
• Raspi

We will have primary/master device deploying PYNQ overlays in the picture below, over Peer to Peer networking (P2P)

Getting Started - Finding Network Devices

Run a script o locate devices on network (Tip: the Ultra96 devices use SEMICONDUCTOR CORPs - name convention to easily identify the names). What this does, it gives you a menu you can select to find all devices on your network, it finds Xilinx Cluster devices, clears the menu and quits if you want. Run and save network.sh in nano text editor

#!/bin/sh
clear
echo "\t `date "+%d/%m/%Y"`"
echo "\t Hostname Machine: `hostname`"
echo "\t user: `whoami` "

echo "\t M A I N - M E N U -- Network seletion \n\r"
echo "\t 1. List All Network \n\r"
echo "\t 2. List Xilinx Cluster Devices \n\r"
echo "\t 3. clear \n\r"
echo "\t 4. Exit \n\r"


while true ; do
        echo -n "\t Please enter an option [ 1 - 4 ] : "
        read opt

        case "$opt" in
                1) echo "Listing All Devices"
        sudo arp-scan --localnet
                ;;

                2) echo "Reading all capstone devices.... "
        sudo arp-scan --interface=wlp2s0 --localnet | grep "REAL"
        ;;

        3) echo "Clear"
        clear
        ;;

                *)echo "Exit" 
        exit ;;
        esac
done

Once you have found the corresponding devices, you can find their Ip addresses. You can make them static such that is easier to find when you reboot | shutdown device.

Package Installation & Deployment (SSH all nodes)

Ubuntu / Debian installation of Cluster SSH

#Install clusterssh - master ssh
sudo apt-get install clusterssh

The following examples will help you make the alterations needed for your ip usernames & ips. Save script cluster.sh in nano text editor.

# cssh command - <username@ip_address_1> <username@ip_address_2> <username@ip_address_3> ..etc

cssh xilinx@10.0.0.85 xilinx@10.0.0.94 xilinx@10.0.0.102 xilinx@10.0.0.251
# xilinx@10.0.0.216

Your file directory should look something like this

Run your cluster --./cluster.sh (Make sure you write in the CSSH[4] box to talk to all units | you can independently choose a window and talk to a selected unit if you prefer)

Device Management (Hostnames)

Upgrade ALL devices

sudo
apt-get update

Change hostname to specific nodes (example is master) [Do not put any underscore or hyphen] SAFETEY use 1 word.

#Master node
sudo nano /etc/hostname
master
#Worker node -- 1 -- Copy and paste process below for each device
sudo nano /etc/hostname
node1        #Edit node on other units

#***********************************************
#Below translate to the different units on cluster 
node2 #- Same process for device 2
node3 #- Same process for device 3
node4 #- Same process for device 4
raspi #- Same process for raspi device
#***********************************************

Network over P2P

This section allows you to now apply what you have just discovered in the environmental setup. You have your hostname changes, you have a way to communicate to each device, now how do we translate our overlays over the network?

A quick description below is what Peer to Peer. Essentially you are pushing data in a magic box, that you can leave on your network, then I can pull it somewhere. This eliminates any local upload and download transmissions. It is secure and based on your setting with your router you can keep the devices private, no port forwarding.

You can transmit any device in the picture to each other, as long as the port is open you and transmit. Think of it as a handshake. An open computer will extend it's hand open and "listening" whilst the computer is transferring will find that port and when it does it will shake its hand and transmit. Once the transmission is complete it will then let go and that busy handshake will drop.

PYNQ - Overlays over P2P

Now we have to have the correct pathing so we can execute all the files necessary. I have copied a single directory with all necessary files below: (this will be linked in the attachment)

capstone_workspace_u96_master/demo/
capstone_workspace_u96_slave/demo

We must first put out our handshake from the slave side. We will execute the path below in CSSH[4]

This is a simple script listening to these different ports below - Make sure you find an open port up to 65, 535.

(The highest TCP port number is 65, 535. The TCP protocol provides 16 bits for the port number, and this is interpreted as an unsigned integer; all values are valid, apart from 0, and so the largest port number is (2^16 - 1) or 65, 535.) :

• port 49664
• port 49665
• port 19132
• port 19133

We have 3 overlays and the demo master sends all files to corresponding ports above. The last overlay attaches from an existing project (refer to THNK-PYNQ design)

• port 49664 - RTL Overlay
• port 49665 - HLS Overlay
• port 19132 - Acceleration Overlay
• port 19133 - PYNQ overlay (extra)

The main page of the code will follow as such. This is Jupyter Notebook's directory structure to then add notebooks run terminal on the browser

Rxcv Code

The code is verbatim on the receive side - all that is changing is the port number.

import socket               # Import socket module

s = socket.socket()         # Create a socket object
host = ''                   # Get local machine name
#**************************
port = 49664                # Reserve a port for your service.
#Change port of other units
#**************************

s.bind((host, port))        # Bind to the port

#path of folder
# path = 'zip_rtl.zip'
path = 'Output_RX_Cluster/u96_k4_Accel.zip'

f = open(path,'wb')
s.listen(5)                 # Now wait for client connection.

while True:
    c, addr = s.accept()     # Establish connection with client.
    print ('Got connection from', addr)
    print ("Receiving...")
    l = c.recv(10240)
    while (l):
        print ("Receiving...")
        f.write(l)
        l = c.recv(10240)
    f.close()

    print ("Done Receiving")
#     c.send('Thank you for connecting')
    c.close()                # Close the connection

Send Command to have ports listen

Txcv Code

This is capturing a file input that is found on master and sending over the data as a packet from the zip file.

import socket                # Import socket module

s = socket.socket()          # Create a socket object
host_1 = knode2         # Set local machine name
port_1 = 49664                 # Reserve a port for your service.

#Connect
s.connect((host_1, port_1))
# s.send("Hello server!")
print('Connected: YES')

#path of folder
# path = 'home/xilinx/jupyter_notebooks/capstone_workspace_u96_master/overlays/zip_rtl.zip'

#open a folder! 
f = open('/home/xilinx/jupyter_notebooks/capstone_workspace_u96_master/demo/bit_files/RTL.zip','rb')
print ('Sending...')
data = f.read(10240)

print('File open: ', f)

while (data):
    print ('data', data)
    s.send(data)
    data = f.read(10240)
f.close()

print("Done Sending")
s.shutdown(socket.SHUT_WR)
print (s.recv(10240))
s.close                     # Close the socket when done

We have contact!

Lets now apply for the remaining overlays

Now we have transmitted successfully our overlays, the next question is how do I open and run my overlays?

Now refer to the last modules and see how each module is constructed for further detail.

• Part 1 - Super96s Cluster - Part 1 - Hackster.io
• Part 2 - Super96s Cluster - Part 2 - Hackster.io
• Part 3 - Super96s Clusters - Part 3 - Hackster.io

K3s - Linux Packages (All devices)

Here is where things maybe a bit tricky... you have to be positive which devices you are running as the [Master Agent - Server] VS [Worker Agent - Client]. If you make a mistake it is all fine, you just have to backtrack your steps, this could become messy. Uninstall if you have to an restart (I did this maybe 20x! It's okay :D )

Master Agent - Server

K3s provides an installation script that is a convenient way to install it as a service on systemd or openrc based systems. This script is available at https://get.k3s.io. To install K3s using this method, just run:

#Run script installation
curl -sfL https://get.k3s.io | sh -

#Server Starter
sudo k3s server &

Worker Agent - Client

To install on worker nodes and add them to the cluster, run the installation script with the K3S_URL and K3S_TOKEN environment variables. Here is an example showing how to join a worker node:

#Run script in cssh window
curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=mynodetoken sh -

Example - Running k3s with your Devices

I want to run a server, and the device I'm using is some type of Linux system (Could be Ultra96 | Could be another computer). You have to have the correct ip address and it has to connect to the server.

Master Agent - Server

#--- Actual Script MASTER - SERVER 
# Replace "myserver" with server/host ip address 

curl -sfL https://get.k3s.io | K3S_URL=https://10.0.0.216:6443 K3S_TOKEN=mynodetoken sh -

Worker Agent - Client

#--- Actual Script WORKER - CLIENT
# Replace "myserver" with server/host ip address 

curl -sfL https://get.k3s.io | K3S_NODE_NAME="node01" K3S_URL="https://10.0.0.216:6443" K3S_TOKEN=

#***********************************************
#Assert IP address for other devices - CHANGE the NAME! 
#node2
curl -sfL https://get.k3s.io | K3S_NODE_NAME="node02" K3S_URL="https://10.0.0.216:6443" K3S_TOKEN="K10bc983c6f089b24f4936c948b39f122a8431c0d3456d148e932ab67f7a67f3607::server:a0999b81fef034caab1b1a48082c513d" sh -
#node3
curl -sfL https://get.k3s.io | K3S_NODE_NAME="node03" K3S_URL="https://10.0.0.216:6443" K3S_TOKEN="K10bc983c6f089b24f4936c948b39f122a8431c0d3456d148e932ab67f7a67f3607::server:a0999b81fef034caab1b1a48082c513d" sh -
#node4
curl -sfL https://get.k3s.io | K3S_NODE_NAME="node04" K3S_URL="https://10.0.0.216:6443" K3S_TOKEN="K10bc983c6f089b24f4936c948b39f122a8431c0d3456d148e932ab67f7a67f3607::server:a0999b81fef034caab1b1a48082c513d" sh -
#raspi
curl -sfL https://get.k3s.io | K3S_NODE_NAME="node05RPI" K3S_URL="https://10.0.0.216:6443" K3S_TOKEN="K10bc983c6f089b24f4936c948b39f122a8431c0d3456d148e932ab67f7a67f3607::server:a0999b81fef034caab1b1a48082c513d" sh -
#***********************************************

There are a lot of Kubernetes commands... since you are using k3s you have to make sure you incorporate k3s at the beginning of each Kubernetes command.

• sudo k3s kubectl get nodes
• sudo k3s kubectl get namespace
• k3s kubectl get pods --all-namespaces\

This is just the simple building blocks of how you can create a system to join them. A proof of concept that you can implement a foundation of Kubernetes on ultra96 edge devices. This also shows you can use devices outside the realm of Xilinx giving more flexible measures for the software engineer. Combining a world involving FPGAs & MCUs.

Deleting nodes on cluster

• kubectl get nodes
• kubectl drain < node-name >--ignore-daemonsets
• kubectl delete node < node-name >

Commands to kill the cluster (either one will work)

• sudo k3s-killall.sh
• /usr/local/bin/k3s-killall.sh

End

Congratulations you have just instantiated your FPGA cluster!!!

Acknowledgments - Special Thanks

Mike Rockel

Adam Taylor

Xilinx <> AMD - ISM Team