EPU - Ethereum Processing Unit

What is this project about?

Everyone running a full Ethereum node needs to process the smart contracts hosted on the block chain. Normally, this is done by emulating the Ethereum Virutal Machine (EVM) in software.

The aim of this project is to build a hardware implementation of the EVM on an FPGA. The result is going to be the Ethereum Processing Unit (EPU), a 256bit processor which does not operate on registers but on a stack.

For whom is this project?

• As far as the design part is concerned, this project is for exotic CPU enthusiasts.
• As far as the hardware part is concerned, it is a project for server modders. The Xilinx Varium C1100 needs a high performance cooling solution. And running Xilinx' software suite requires a custom built work station.
• You want to use this project? Then you are a solo miner, mining pool operator or a validator on an EVM powered block chain.

Set up the Work Station Hardware

Choosing quite quiet components is OK to cool your workstation running Xilinx software but not for the machine hosting the Varium C1100. If you want to host the card in the same PC case, read the chapter "The Cooling System gets Beefed up" before proceeding to physical installs.

Component choice:

• a fanless graphics card
• a fanless power supply (500 W)
• 3 x 140 mm fans that come with the Fractal Design case
• 1 x 120 mm Noctua CPU cooler

The RAMs do not need extra cooling with my specific BIOS setup (3200 MHz, 1.2 V). NB: I thought of mounting the CPU fan on a shroud with some air escapes towards the RAM bars. But this is not necessary.

For the fanless power supply, we need a positive pressure inside the case. This means that all the three Fractal fans are mounted on the front bezel side. The top and bottom meshes are covered with paper (white areas in photo below) in order to guide the air flow from front to back.

For connecting the fans, untie all cables on the left side in the photo below.. The fan cables are attached using the same ties to avoid their dangling around during transport. The fan hub gets power from SATA. Although the SSD is 2.5", I mounted it in 3.5" a bay. There, it won't get heated up by the main board.

There is no shroud required between the CPU cooler and the back of the case.

The airflow is sufficient. Only few hot spots occur at power converter components on the main board and the graphics card.

As you see, the Varium C1100 did not arrive yet at the time of writing. It must not be operated in an airflow PC case, but in a server case with high static pressure. If you already have your Varium card, do not connect it yet.

Set up the Work Station BIOS

If your mass storage device already has a running OS, boot it before tweaking the BIOS. You can download the BIOS update and save it on the EFI boot partition (as root). Then, reboot your machine into the BIOS and install its update. If your mass storage device has no bootable OS, you need another computer from which you can store the update file on a USB drive.

You will now see that your RAM is not running at the rated speed. Let's fix that. Notice: my system got unstable when setting the usage profile to Workstation/Gaming in the BIOS. So do not touch this setting and go for some manual work:

• Press F7 to switch to advanced mode
• Get timings by enabling D.O.C.P. standard
• The BIOS sets now the voltage of your RAM too high (it is marked in yellow, scroll down). Reduce it to 1.2V, the standard setting. This will avoid your RAM bars running too hot and saves energy. I had no stability issues at 1.2 V up to now.

Install the OS

Download your favorite flavour of Linux Mint from https://www.linuxmint.com/download.php

Copy the *.iso file to a USB stick that is empty or has only data that can be irrecoverably overwritten.

Plug in your USB stick into the computer on which you downloaded the *.iso file. Now make sure whether your USB stick is on /dev/sdb or anything else.

dmesg

If you have one physical hard disk and one USB stick drive, this should be the case. If not beware to adapt the following commands, otherwise your secondary drive will suffer from data loss!

cd ~/Downloads
dd if=linuxmint-20.2-cinnamon-64bit.iso of=/dev/sd<driveletter from dmesg>
sync

The of= parameter is something like /dev/sdb, not /dev/sdb1. Yes, you overwrite the MBR (master boot record), partition table and everything.

Boot your machine from the USB stick.

It will boot a live system with an install icon looking lice a CD. Double click that one.

During install, tick the "... all ... codecs ..." check box. Terminate the install and reboot.

In Linux Mint, the "Welcome" assistant suggests under "First Steps" a "Driver Manager", which in a few mouse clicks identifies your Nvidia graphics card and installs appropriate drivers. One reboot later, you run on the proprietary nvidia drivers.

Other reasons to choose Mint: Set up an English system in a non-English speaking country (Debian and Ubuntu are rather bad at that), and Cinnamon as a window manager.

Configure Firefox to play DRM infested contents. Combining this with the codecs, this allows you playing Xilinx videos on your Linux box.

Install Vitis and prerequisites

If you plan to install Vitis on any Ubuntu based Linux, make sure to install libtinfo5 before starting the installer.

https://support.xilinx.com/s/article/76616?language=en_US

in short: be superuser, update your system and install libtinfo5. Some other installs will be handy or needed later:

sudo su
apt update
apt upgrade
apt install libtinfo5
apt install cmake
apt install openssh-server
apt install git
apt install build-essential
apt install opencl-headers
apt install xsensors

cmake will be used later to build the blockchain acceleration mining demo. openssh-server allows you to log into your workstation from another computer using ssh, or to copy the Vitis installer from another computer using scp.

Make sure to disable journaling on the filesystem on which you are going to install Vitis. Doing so may be easiest if you have a separate 300GB partition mounted under /tools.

This time I installed the entire system including Vitis on an XFS partition with logbsize=256k and only tick in the unified installer what is needed for this project.

If you use the xfs file system, edit /etc/fstab and replace

UUID=......     /     xfs     defaults    0    0
/swapfile      none            swap    sw              0       0

by

UUID=......     /     xfs     logbsize=256k    0    0
#/swapfile      none            swap    sw              0       0

and remove all swap partitions and swap files. We do not want to swap on an SSD drive in a 128 GB RAM system.

Reboot your system (yeah, I know that there are alternatives using mount and swapoff -- but let's keep things simple).

Now it's time to download the Xilinx unified installer:

https://www.xilinx.com/member/forms/download/xef.html?filename=Xilinx_Unified_2021.2_1021_0703_Lin64.bin

Or a larger choice: https://www.xilinx.com/support/download.html

Still as root, we change to the download directory and run

chmod 555 ./Xilinx_Unified_2021.2_1021_0703_Lin64.bin
./Xilinx_Unified_2021.2_1021_0703_Lin64.bin

Leave the radio button on the top one ("Vitis"). It will also install Vivado. Yes, Vitis depends on Vivado. From the components, remove all you do not need. The U55 on the Varium card is an UltraScale+ device.

Leave the following items checked:

• Vitis Unified Software Platform (with all 3 components)
• DocNav (always good to have documentation)
• ... Alveo and edge accel... (some components of the Varium are Alveo compatible)
• Devices: UltraScale+

Uncheck the following unless used for other projects:

• Vitis Model Composer (No, I don't have Matlab)
• Devices for Kria SOM (if this one contains PetaLinux, we can save some Gigabytes)
• Devices for Custom Platforms: All except UltraScale+

Leave Engineering Samples unchecked.

The installation process takes several hours.

Afterwards, some work is needed to avoid running Vivado with root privileges:

https://support.xilinx.com/s/feed/0D52E00006hpUGrSAM

And for building Xilinx DMA:

https://github.com/Xilinx/dma_ip_drivers/issues/39

Hooray, the Varium card arrived

An important date (2021-12-20) in the history of this project: The physical Varium C1100 card enters the scene. A big thanks to Xilinx!

The package contains nothing else than the Varium card. All software is on the web site. No cables provided.

The card itself looks like a smart NIC. But we all know: it's a very smart NIC with HBM. It has 8 MAC addresses for the two QSFP-28 ports. Made in Malaysia.

The programming port connects to Micro-USB.

The card is heavy. It's all about cooling. There is also a metallic backplate. Anyboy knows what J37 is good for? I2C? I would like to have a PWM output for the fan. Maybe I could also abuse the Alveo connector for that.

... And a Hackster.io sticker. Cool.

The mechanical install of the Varium card

Option 1: vertical slot

For this project, I bought a computer case with vertical GPU slot and solid side panel. My hope was to have more options for shrouding, especially since shrouding near the mother board is expected to be problematic.

Problem shown: the front USB 3 cable would have been better seated next to the power supply connector.

----

Option 2: Bottom slot

The bottom slot of this main board only has 8 PCIe lanes. Shrouding next to the main board with a temporary paper shroud failed. If you use this slot, attach your shroud directly to the Varium card.

The cooling system gets beefed up

Option 1: Old 80mm ProLiant server fan + shroud

The shroud is attached directly to the Varium card. So, it is OK for bottom slot and vertical slot mount. Only the PCIe AUX power cable passes the shroud.

The fan runs at full speed and makes a rather unpleasant noise. But it delivers 10 mm H2O of static pressure (operating on 12VDC). Make sure your probing hose is perpendicular to the air flow.

The server fan is too loud when operated on 12VDC. Undervolting it to 5VDC (also readily available on a Molex connector) does a good compromise between noise nuisance and sufficient cooling of the Varium card at stock settings (i.e. no overclocking).

----

Option 2: Sanyo Denki high static pressure fan

(!) Hazardous area (!) No kidding. Touch a running PC fan and it just stops. Touch a running server fan and you bleed (first hand experience made). Touch a running industrial grade fan and ... No. Just don't. There are no warning labels on the Sanyo Denki product.

No free running tabletop tests! This fan has got too high thrust. Mounted directly into the PC case for first energizing test, of course with protective grid:

Wiring: red: +48V, black: 0V, brown: PWM (connect to 0V for 0% PWM!), yellow: tacho (not connected).

The Sanyo Denki makes a nice soft start. No power surge during power up. Even at 0%PWM, this copter builds up a solid static pressure. But it is too loud. it consumes around 7-14W. These values are badly reproducible on my bad Watt meter, which has a resolution of 7W.

The next test was to power up the copter to 100% PWM. I shut down the computer. This way, even in case of a LOCA (loss of cooling agent), I would not risk a core meltdown in my brand new Varium reactor sponsored by Xilinx.

The Tektronix power supply is OK for 0%PWM. But for 100%PWM I used a power supply I designed for an audio amp project, I tried to speed up the copter to full power. Disconnecting the brown wire from the black one is enough for doing so. Wearing safety goggles and ear protection, of course. Comment of my wife? Frightening! She is right: The Sanyo Denki copter slowly ramped up from 3000 RPM to below 15000 RPM according to data sheet. But my power supply went into current limit (1.5 A) and power consumption was at 77W, below the rated 98W. The RND components fan protection grids are clearly not designed for that throughput. The air flow gets too turbulent and all energy is absorbed there instead of thrusting air through the Varium card or building up pressure in the PC case.

If you really want to sue the Sanyo Denki copter in spite of its noise, shroud it directly to the Varium card. Maybe you want even to build a wind channel that also cools the back and front plate. On the intake side, add an aerodynamic safety shroud. Happy overclocking!

The copter will be low noise modded in a later project.

Meanwhile, my Hackster's lab is a bio hazard area since I caught COVID.

----

Option 3: three 140mm fractal fans in series

Finally an acceptable solution for an idle system.

Take the fractal fans out of your case, line them up so that all cables are on the same side and the air flow goes into the same direction. Then, insert the BBQ skewers (max diameter 3 mm) with their tips pointing against the air flow into the fans' mounting holes and the spacers:

Now, roll adhesive tape on the BBQ skewers (stick a little tape on the skewer and start rotating the skewer until the tape layer is so thick that it won't pass the mounting hole anymore). Profit from the thinner end being a little inside the assembly of the first fan.

Make sure that the fans are stacked tightly together.

Cut the skewers inside the first fan assembly on the intake side and outside of the last fan assembly. Cut a paper that is about 600mm long and as wide as the fan stack. Stick it to two spacers near the wire and wind it around the fan stack. Add adhesive tape to make the cable passage air tight. Mount it just above the power supply shroud.

You now have two options:

a) Continue a shroud to the Varium card and add a few more fans for the rest of the PC

b) Cover the remaining of the front plate with adhesive tape so that no air can escape to the front. The back side of the PC case is to be masked partially with adhesive tape so that enough air is forced through the Varium card. Tweak the path of least resistance.

I've chosen option b).

Not a beauty but OK for a proof of concept.

We have a few mm of static pressure at an acceptable noise level.

As long as the system is idle, temperatures are OK, too.

As long as the Varium is idle on the golden or xilinx_u55n_gen3x4_xdma_base_1, power can be applied without proper cooling. The card consumes between a little less than 10W and 14W, depending on the firmware loaded.

----

Conclusion:

Cooling the Varum card at acceptable office noise levels is a challenge.

The accepted solution is:

• Cool the entire PC case with 3 stacked fans (option 3), but apparently the fan configuration shown in the section "Set up the workstation hardware" would be OK too.
• Cool the Varium card with an undervolted (5V instead of 12V) server fan.

A speed control would be nice since it allows overclocking.

Giving that one a try during a later project: https://github.com/neffs/fan_controller

A minus point to Asus for not enabling me to use a chassis fan header for controlling the fan shrouded to the Varium card.

Temperature monitoring

If you run

xsensors

you get an idea of some temperatures. There are two bugs

• No distinction between 0 and NULL
• HBM temperatures not reported (I manually added red color using gimp).

Reverting to Golden and re-flashing the shell miraculously solved the HBM temperature readout problem.

Do not run "sensors" or "xsensors" when anything else than the xilinx_u55n_gen3x4_xdma_base_1 image is flashed. Under Team Red Miner, these commands hang.

Trying to get the mining demo running

Adam Wolnikowski started mining one week earlier than I received the card. https://www.hackster.io/contests/xilinxadaptivecomputing2021/discussion/posts/8978

Trying to catch up ;-)

In order to get first signs of life of your Varium card, use the lspci and lsusb commands:

lspci | grep Xilinx

should yield a line like

09:00.0 Processing accelerators: Xilinx Corporation Device 5058

Do not pay any attention for the numbers at the beginning of this line. They depend on your main board and in which slot you inserted the Varium C1100 card.

Now, connect a Micro USB cable between the service connector of the Varium card (accessible from outside the PC case) and any USB slot on your main board.

Then, enter the following command:

lsusb | grep UART

It should yield an output like

Bus 004 Device 004: ID 0403:6011 Future Technology Devices International, Ltd FT4232H Quad HS USB-UART/FIFO IC

Again, do not pay any attention to Bus and Device number for now. They depend on your main board and which USB connector you used. If you have many UARTs connected, use a different keyword after the "grep" command. In case of doubt, run lsusb twice: once with the cable connected, once with the cable disconnected. The difference between the output is what counts.

This USB connection is used for the "out of band" communication with the Varium card. You radio this way to the 2nd pilot (called "satellite controller" by Xilinx) of the Varium card. UG1363 is not Varium specific but can give you an idea. Temperatures can be read out via I2C. Yeah, I want to control fan speed later.

First, let's download the Xilinx minimg demo. Avoid being superuser at this point.

Make/change to the directory where you want to store the project. The following command will create a directory "blockchainacceleration" and download the project files:

git clone https://github.com/Xilinx/blockchainacceleration.git

Now, you are ready to follow the Xilinx video tutorials:

Part 1: https://www.xilinx.com/video/accelerator-card/installing-varium-blockchain-accelerator-cards-for-eth-mining.html

Key takeaways:

• Set fans to max speed in the BIOS
• Mind the PCIe lanes. Some slots of your main board may be stripped down (look like 16 lanes but are 8 lanes).
• Connect a PCIe AUX power (=GPU power) to your Varium card. Make sure not to buy/connect CPU/Mobo connectors. They look very similar but have an incompatible pinout.
• Have a high static pressure (server fans -- simple workstation fans are not enough)

I now intentionally skip Part 2, since part 3 contains prerequisites for part 2:

Part 3: https://www.xilinx.com/video/accelerator-card/running-ethereumminer-on-dell-servers-varium-blockchain-accelerator.html

In case your /opt/xilinx/... folder does not exist, it is now time to install the runtime:

• https://www.xilinx.com/products/accelerators/varium/c1100.html#gettingstarted
• in my case: https://www.xilinx.com/bin/public/openDownload?filename=xrt_202110.2.11.634_20.04-amd64-xrt.deb

Change to the ~/Downloads directory, then (your version may differ):

sudo apt install ./xrt_202110.2.11.634_20.04-amd64-xrt.deb

Also from the getting started page, let's download (these or any newer version):

• https://www.xilinx.com/bin/public/openDownload?filename=xilinx-u55n-gen3x4-xdma-all_1-3191732.deb.tar.gz
• https://www.xilinx.com/member/forms/download/eula-xef.html?filename=xilinx-u55n-gen3x4-xdma-1-202110-1-dev_1-3236984_all.deb

Let's decompress the tar archive:

tar xfvz xilinx-u55n-gen3x4-xdma-all_1-3191732.deb.tar.gz

Note: in the video, the deb packages are installed one by one. On my system, this did not work due to dependency issues. Put all deb files into the same apt install command. Problem solved:

sudo apt install ./xilinx-u55n-gen3x4-xdma-validate_1-3191757_all.deb ./xilinx-cmc-u55_1.5.16-2.3143933_all.deb ./xilinx-sc-fw-u55_7.1.12-1.ba326cb_all.deb xilinx-u55n-gen3x4-xdma-base_1-3191732_all.deb ./xilinx-u55n-gen3x4-xdma-1-202110-1-dev_1-3236984_all.deb

As advised:

sudo /opt/xilinx/xrt/bin/xbmgmt flash --update --shell xilinx_u55n_gen3x4_xdma_base_1

Now, it's time to power cycle the machine hosting the Varium card.

Welcome back.

Let's se whether the upgrade worked:

lspci | grep Xilinx
09:00.0 Processing accelerators: Xilinx Corporation Device 5058
09:00.1 Processing accelerators: Xilinx Corporation Device 5059

Yeah, we have one more line. Re-run the same command as above:

sudo /opt/xilinx/xrt/bin/xbmgmt flash --update --shell xilinx_u55n_gen3x4_xdma_base_1

It will now flash the SC.

There is a deprecated warning. If things no more work, look here.

Now comes the validation part:

sudo lspci -vd 10ee:
sudo /opt/xilinx/xrt/bin/xbmgmt flash --scan

Make sure you can see the scan results and replace the 09:00 in the following command accordingly:

sudo /opt/xilinx/xrt/bin/xbutil validate -d 0000:09:00.1 --verbose

Yeah, for the first time a real test on the card.

Some messages are strange though:

Test 1 [0000:09:00.1]     : Aux connection 

    Description           : Check if auxiliary power is connected
    Details               : Aux power connector is not available on this board
    Test Status           : [SKIPPED]

I have AUX power connected.

Part 2: https://www.xilinx.com/video/accelerator-card/installing-eth-mining-software-varium-blockchain-accelerator-card.html

Spoiler alert: I did not succeed. If you want a running system, skip to the next chapter.

Here are the files to download (we can't get them with curl since these URLs are log-in walled):

• https://www.xilinx.com/member/forms/download/xef.html?filename=dag_gen_kernel.xclbin
• https://www.xilinx.com/member/forms/download/xef.html?filename=ethash_kernel.xclbin

Now, let's set up the environment variables

source /opt/xilinx/xrt/setup.sh

I ran into problems with cmake, probably because I set up Vitis. Make sure a

cmake --version

yields a version >= 3.5

./build.sh build_host

Fails on missing CUDA toolkit. Since I do not want to mine on a fan-less graphics card, I disabled CUDA:

In the build.sh file change line 146, to

cmake .. -DETHASHCUDA=OFF

Thanks to Fouad for spotting that one.

source /opt/xilinx/xrt/setup.sh
./build.sh build_host

[100%] Built target ethminer

Yeah!

If you do not have an ETH wallet, create one. I took WallETH from https://f-droid.org/

Then start editing your startup script:

nano mine.sh

The script looks like

source /opt/xilinx/xrt/setup.sh
./build.sh mine 0x295805bbbaf99ec63183bfe5078c8bc34203e7 xiliminix eu1.xxxxxmine.org:4444

where you replace the 0x... by the public key of your wallet, xiliminix by your worker name and eu1.xxxxxmine.org:4444 by hostname:port of your mining pool. Do not use ethermine.org. They blacklist you very quickly on unsuccessful attempts.

Now, let's make this script executable and run it:

chmod 755 mine.sh
./mine.sh

Did you really expect that this was going to work?

On my system, ethminer got built in my home directory. So stay where build.sh is and

mv ~/ethminer .
./mine.sh

The answers are:

Unrecognized platform Xilinx
Subscription failed : Invalid user provided

Sounds more promising. I apparently have to connect my WallETH with ethermine.org. This can be done with the "connect wallet" option on the top right of the ethermine.org website.

And IIRC, I did not flash the DAG and mining kernel on the Varium card yet.

xbutil program --device 0000:09:00.1 --user dag_gen_kernel.xclbin
xbutil program --device 0000:09:00.1 --user ethash_kernel.xclbin

I hope they don't overwrite each other.

Apparently, ethermine banned me. Using eu-eth.hiveon.net:4444 instead.

First tries to mine with my graphics card, then...

SIGSEGV

sudo su
source /opt/xilinx/xrt/setup.sh
xbmgmt program -d 0000:09:00.0 --revert-to-golden

Tells me to power cycle. Maybe that's what I missed after programming the user partitions.

Another thing I did not verify is whether I built daggen for the right platform.

A bunch of thermal imagery -- Team Red Miner in Action

Originally specialized on AMD/OpenCL miners, Team Red Miner turns your Varium card into an ETH PoW worker easily.

Create a new user, make it member of the group dialout and reboot your machine. Make sure the trm user can access ttyUSB<n>. Otherwise, you can run Tea Red miner as root, but that's bad practice.

Here are the temperatures (cooling with undervolted (5V instead of 12V) server fan):

Team Red Miner supports overclocking the card. But that needs better cooling. This can be provided by switching the server fan to 12V:

sudo su
source /opt/xilinx/xrt/setup.sh
./teamredminer --eth_dag_slowdown=0 --fpga_clk_core=625 --fpga_clk_mem=1250 -a ethash -o stratum+tcp://eu-eth.hiveon.net:4444 -u 0x2a13f15b04f7b8d6319e9ed56521f2c2adcaf324 -p x

Adapt the parameters to suit your needs. Otherwise, it will mine into my wallet ;-).

Your maximum clock values may differ from mine, depending on cooling and manufacturing tolerances.

The hot spot is pushed further away from the fan. Bad for the power converters.

The fan speed is more decisive over the temperatures than the overclocking speed. To the right of the USB connector, I cut away some parts of the rear plate in order to facilitate air flow. On the thermal image, we can clearly see the main cooler. Do not cut away the rear plate above the QSFP28 connectors. We need more air flow below. Mind the path of least resistance for the air.

Blinky - the Hello World of FPGA

Team Red Miner is at the time of writing one of the rare applications that run out of the box on the Varium C1100 card. But when you buy an FPGA, you usually want to run your own designs on it. And a blinking led is the first proof you can produce that the entire design flow works.

Luckily, @Quarky helped me to get Blinky running on the Varium.

Change to your projects directory and download Quarky's code. There is more than blinky to it, but let's make a first baby step:

git clone https://github.com/Quarky93/warpshell.git

Follow the instructions in the README.md file concerning Blinky.

Once you programmed it, one of the SQFP28 port status LEDs starts blinking, the others stay on. The low frequency is my fault ;-). Increasing it is left as an exercise to the reader.

Getting the DMA demos running is somewhat more challenging because you have to build the xdma driver:

Change to the directory where you build your projects. Then

source /tools/Xilinx/Vivado/2021.2/settings64.sh
git checkout https://github.com/Xilinx/dma_ip_drivers.git

and create an SSL key according to https://github.com/Xilinx/dma_ip_drivers/issues/39

Now, you can start the build process for the xdma drivers. Refer to the README file. Then,

modprobe xdma
lsmod | grep xdma

will show whether your build was successful. If

lspci | grep x

shows your Varium card as 9041, then you're all set and you can try Quarky's sensor and DMA demo. If your card shows as 5058, you still have the Xilinx runtime loaded on your card. Program Quarky's shell using Vivado's hardware manager. If that does not change the lspci 5058 to 9041, do a warm reboot (i.e. without power cycle).

Once some /dev/xdma* files show up, the xdma drivers are built correctly and your card is programmed correctly.

Notice: If you update your kernel, you have to rebuild your xdma driver. Do a make clean before re-creating an SSH key and make install.

Configuring Vivado for the Varium C1100 the GUI way

When starting Vivado, you will miss the Varium C1100 among the boards for which you start the design. There are two workarounds for this problem:

• the tcl console way: UG1526 page 9 (download from https://www.xilinx.com/member/varium-vivado.html )
• the GUI way:

Start vivado

source /tools/Xilinx/Vivado/2021.2/settings64.sh
vivado

• Create a new RTL project
• Choose any Ultrascale+ board
• Go to the IP catalog
• Disable the "hide disabled and incompatible IPs" button
• Select "Card Management Solution Subsystem" Version 4 (try one of the two, the version will appear in the panel below, or scroll the table to the right)
• Right-click on it and choose "Compatible families"
• Choose XCU55N-FSVH2892-2L-E (use the search feature: Magnifying glass) for the Varium C1100 (you can find this part number in DS1003 and UG1526). You also can find cards like the U55N, U55C, SN1022...
• Click on "Switch part"
• Confirm switching the project part
• Include the C1100_xdc_1v00.xdc file (or any newer version) into the design constraints

You can download the XDC file from this page: https://www.xilinx.com/products/accelerators/varium/c1100.html#vivado

Now, you are ready to go for creating your new design.

Using MicroBlaze as a "Service Processor"

I first thought of building an extended EPU with

• supervisor mode (some additional I/O capabilities for loading the contracts and their state into memory and returning the new state
• execution mode (executing the smart contracts like EVM)

This is less probable to succeed than using MicroBlaze for doing the supervisor mode tasks above. We do not need a 256bit ALU and stack engine for loading jobs into memory. The idea is also not to use PCIe/DMA in the first version, but MicroUSB UART. Like this, I can avoid rebooting the host for getting PCIe right. We probably also can avoid writing driver software. Performance may be lower than emulating the EVM on the main CPU. If so, DMA can still be introduced later.

Please note that all documentation referring to SDK is outdated. We now use Vitis for implementing software on th MicroBlaze.

https://www.xilinx.com/support/documentation/quick_start/microblaze-quick-start-guide-with-vitis.pdf

Also search for MircroBlaze Hello World. Beware of old SDK workflows.

Depending on the capabilities of Vitis, it also could be possible to take a "fake before make" approach, i.e. implement the EPU on the microBlaze and replace it step by step by the real hardware EPU.

Trying to get geth running

Change to your project directory and avoid being superuser, then:

git clone https://github.com/ethereum/go-ethereum.git
sudo apt install golang
cd go-ethereum
make all

Did you really expect a smooth build process? Version 1.13.8 installed by the commands above throws the following error:

build github.com/ethereum/go-ethereum/cmd/clef: cannot load hash/maphash: malformed module path "hash/maphash": missing dot in first path element

An update of golang could fix that? let's try. Some instructions here look good.

sudo add-apt-repository ppa:longsleep/golang-backports
sudo apt update
sudo apt install golang-go
sudo apt autoremove

let's try again:

make all

The README does not tell us how to make the executables accessible to bash. I found that at the bottom of this page.

cd build/bin

Wow, there is even an evm executable. What does the README say?

./evm --code 60ff60ff --debug run

output is:

0x
#### TRACE ####
PUSH1           pc=00000000 gas=10000000000 cost=3

PUSH1           pc=00000002 gas=9999999997 cost=3
Stack:
00000000  0xff

STOP            pc=00000004 gas=9999999994 cost=0
Stack:
00000000  0xff
00000001  0xff

#### LOGS ####

Neat! That one still runs on the CPU. For a "hello world" on the EVM, it is great to have this tool.

./geth --goerli console

starts syncing the testnet. That will be a benchmark for how effective EPU/EVM acceleration is. But bugging peers interfere with the timing. Geth is already multi-threaded. Good. -- but not the EVM. Could make life easier.

Missed a --syncmode full option. That should have called the EVM.

Reverse engineering the existing geth EVM code

https://github.com/ethereum/go-ethereum/tree/master/core/vm

More precisely

https://github.com/ethereum/go-ethereum/blob/master/core/vm/interpreter.go

Here, we see

func (in *EVMInterpreter) Run(contract *Contract, input []byte, readOnly bool) (ret []byte, err error) {

So, the strategy is to set up a DMA sending contract *Contract, input []byte, readOnly bool to the Varium card, wait until processing is done and fetch ret []byte, err error by DMA.

The interpreter main loop will be effectively replaced by the EPU. The rest of the code in interpreter.go is needed to execute precompiled contracts that neither run on EVM nor EPU.

Now, I'm going to do HW implementation related changes on my geth fork: https://github.com/stonux/go-ethereum.git

If you want your own copy of it, change to your local work directory. Beware if there is already a go-ethereum there. I do not know how well IDEs handle multiple forks.

git clone https://github.com/stonux/go-ethereum.git

How weird is the whole thing going to be?

Remember that Ethereum wants to be ASIC resistant. This is toxic for a hardware implementation of the EVM. Fee schedules, OpCodes... change from one fork to another. If we do a full sync of a full node, we are going to have to re-program the FPGA for every fork, or implement a EPU that takes a fork ID as an argument and adapt instruction decode and gas metering accordingly (maybe in a later version).