AI Marketplace and Connected Picture Frame

Overview

We are living in a world that is increasingly becoming smart. From self-driving cars, to intelligent stores, AI's are being developed to play video games, recognize skin cancer, and create art! As a relevant example, not too long ago, the auction house Christie's sold an AI-generated artwork for $432,500!

In a world with AI artists we need a way to request new artwork from the AI's and a way to display it for humans to view. This is the purpose of IOTA Canvas and AI Marketplace.

IOTA Canvas

IOTA Canvas is a smart picture frame which automatically refreshes the art it displays on a periodic basis so its viewers see new art every so often. IOTA Canvas has the following features:

• Easy configuration
• Configurable screen auto-off to save power overnight
• Hardware buttons for showing the configuration instructions, skipping an undesirable artwork, and liking an artwork
• On-device automated management of IOTA cryptocurrency

The ultimate vision for IOTA Canvas is for a picture frame which learns its viewer's preferences for art over time and selects the AI artists that cater to those preferences. It will be a full-fledged Internet of Things (IoT) device that allows easy WiFi configuration, operation, and over-the-air updates. To enable all these features, including requesting new artwork from an AI artist, IOTA Canvas communicates with an AI Marketplace.

AIMarketplace

AI Marketplace is a service that manages AI content creators and handles negotiations with clients like IOTA Canvas. It allows people to upload an AI creator which can then respond to requests for new art.

The goal for AI Marketplace is to enable AI content creators of all types to connect with content consumers. This project showcases one such relationship with AI-generated art displayed on a smart picture frame. However, AI Marketplace could also eventually serve unique AI-created recipes, songs, clothing patterns, and short stories tailored to each individual consumer's tastes.

Now the question is, how can consumers pay for their content? The AI Marketplace needs to charge a fee to the consumer for each piece of content created to offset its costs. The AI Marketplace must pay for the servers it runs on, the electricity it uses to power the AI's, and may need to pay an additional surcharge to some of the human's that developed the AI's. This is where IOTA comes in.

ANoteonIOTA

Built to serve the web of internet of things, IOTA is a cryptocurrency that allows trustless sub-cent payments between machines and between humans. It is perfect for IOTA Canvas and AI Marketplace because it enables IOTA Canvas to pay the AI Marketplace for the art generated by the AI without any human involvement. For more information on IOTA, see the IOTA website.

Using IOTA Canvas

Once IOTA Canvas has been built following the instructions in the next section, using IOTA Canvas is simple. When the picture frame first starts up it will display a website address which you can navigate to on your phone to fund it's wallet with IOTA so it can pay for new artwork. Get test tokens from the IOTA devnet faucet to experiment with.

Additionally, the settings website allows you to see the IOTA balance and adjust a number of other settings, such as which marketplace it connects to, how often the artwork changes, the GPIO pins for the buttons, and when the display turns off.

Overtime the picture frame will show various artworks from different AI artists. If you see one you dislike, press the skip button. If you want to change the settings, press the setup button, which will display the address of the settings website.

How IOTA Canvas Works

IOTA Canvas has three main parts, each run by a different thread. The display, which handles updating the images and text on the screen, turning the screen on and off, resizing images if necessary; the settings web server which manages updating the settings and communicating with the user; and the main thread which handles almost everything else, reacting to button presses, checking if it is time to refresh the artwork, telling the display to turn on or off based on the settings, etc.

One more thread handles the actual refreshing of the artwork. An extra thread was chosen for this because it is a long-running task. The process for refreshing the artwork consists of five main steps:

1. Ask the AI Marketplace for a list of all of its artists.

2. Select the artist that best matches the human owner's preferences and budget.

3. Request that the AI Marketplace commission a new piece of art from the selected AI. The AI Marketplace will reply with an IOTA address and a secret key.

4. Pay the requested amount to the given IOTA address.

5. Repeatedly check with the AI Marketplace every 30 seconds if the commissioned artwork is ready.

6. When the artwork is ready, download it, and display it.

Building IOTA Canvas

Putting IOTA Canvas together is fairly straightforward. Begin by hooking up the Raspberry Pi to your monitor and testing that it is working correctly. Solder wires to the buttons and wire them up as shown in the attached wiring schematic. Ensure that the wires are long enough to place the buttons where you would like to, usually on the edge of the frame is best. Remove the case from your monitor and place it inside a nice frame (see cover photo).

With the hardware taken care of, the next step is to get the software running. The IOTA Canvas software only runs on Python 3.6, 3.7 so first check which version of Python your Pi is running and if necessary, install one of the versions above. The software also uses pipenv to manage dependencies and its virtual environment, which is recommended by the Python Software Foundation. If you don't already have pipenv, install it with:

$ pip install pipenv

Next, download the IOTA Canvas software by cloning it to /usr/bin and install its dependencies. Note: if you don't place the project in /usr/bin, you will need to edit the WorkingDirectory in the iotacanvas.service file.

$ cd /usr/bin$ git clone https://gitlab.com/iota-lab/iota-canvas.git
$ cd iota-canvas$ pipenv install -e iotacanvas

Check that everything works by running the main file:

$ pipenv run python iotacanvas/main.py

This will initialize the application, generate an IOTA seed, and save everything to a settings.conf file which can be used to load the settings later. You can exit the application by pressing the escape key.

Now that IOTA Canvas is setup and working, we want to make it automatically startup when the Raspberry Pi boots. We can do this using systemd which is a service manager for Linux. The systemd service file has already been created in the iota-canvas repository. Run the following to install the service and start it on boot:

$ sudo cp iotacanvas.service /etc/systemd/system/iotacanvas.service
$ sudo chmod 644 /etc/systemd/system/iotacanvas.service
$ sudo systemctl enable iotacanvas

Reboot the Pi and check that everything is working as expected.

Short Overview of IOTA Functions

Upon initialization, IOTA Canvas generates and stores a seed (a unique string of 81 letters + the digit '9'). The seed is essentially the password for the IOTA wallet. IOTA Canvas takes care of managing the seed and addresses so the user doesn't have to worry about losing it or accidentally revealing it to someone. The seed is used to initialize the IOTA APIs that interact with the IOTA Tangle.

Accessing the IOTA Tangle and handling IOTA is relatively easy because of the libraries put together by the IOTA Foundation and community. As an example, checking the balance of a wallet that you have the seed for is as simple as the following API call, where the stop_index is optional, but is used to ensure that the balance from all addresses up to stop_index are added up. The pyota library (as the Python IOTA library is called) will check all the addresses starting from 0 up to the stop_index, where each address is associated with one index (in general). If the stop_index is not provided, the pyota library will check up to the first address with no transactions referencing it. If you have skipped indices when generating addresses, then you must provide the correct stop_index in order to get an accurate assessment of the balance available with the given seed.

def get_balance(iotaapi):
    """Returns the iota balance associated with the settings seed"""
    # get the stop index which is saved in the user_settings
    stop_index = user_settings.get(['iota', 'addr_index'])
    # get the account details from each address up to the stop_index
    return iotaapi.get_account_data(stop=stop_index)['balance']

Sending IOTA is a little bit more complicated, but still relatively easy. A slightly simplified version of the code in IOTA Canvas is shown below. First the transaction is assembled, and then it is sent to the IOTA Tangle for confirmation. A change address is used to store extra IOTA beyond the amount sent. The min_weight_magnitude (MWM) parameter is used to specify how difficult the Proof of Work should be. The harder the Proof of Work, the longer it will take, but the IOTA devnet requires a MWM of 9 and the mainnet requires an MWM of 14. Proof of Work is a general name for the mathematical problems the machine must solve in order to prove that transactions are valid. The depth parameter controls how deep into the IOTA Tangle to look for an attachment point for the new bundle of transactions.

def send_iota(iotaapi, amount, destination_address):
    """Sends the specified amount of Iota to the specified address"""
    # assemble the transaction with the destination address and amount to send
    tx = iota.ProposedTransaction(
        address=iota.Address(destination_address),
        tag=iota.Tag('VALUETX'),
        value=amount
    )
    # generate a new address to send the extra IOTA to
    change_address = generate_address()
    tx = iotaapi.prepare_transfer(
        transfers=[tx],
        change_address=change_address
    )
    # do the proof of work and send the transaction
    iotaapi.send_trytes(tx['trytes'], depth=3, min_weight_magnitude=14)

You can see the full functions here in the IOTA Canvas repository.

AI Artist Details

The current state-of-the-art in AI art generation uses variations on what is known as a Generative Adversarial Network (GAN). GANs consist of two parts, a generator and a discriminator. The generator tries to create a plausible piece of art after being initialized with random noise. The discriminator's job is simply to try and figure out if an image it is shown is real, or fake (fabricated by the generator). The discriminator's results (whether an image is fake or not) are then used to teach the generator how to make better art. So the generator is constantly trying to fool the discriminator, while the discriminator tries to distinguish the fakes from the originals.

The kinds of real images that are shown to the discriminator are the ones that the generator learns to generate. So by using different collections of art, we can make generators that produce art in different genres or styles. In fact, GANs are not limited to producing art. If they are trained enough on real photos, they can learn to imitate them. Checkout thispersondoesnotexist.com for a cool demonstration of a realistic fictional person generated by a GAN.

GANs take a long time to train, so for this project I was only able to train one. I chose to create a GAN that produces landscape art. See below for a short video that demonstrates how this GAN learned, starting with producing images of a single color and then eventually creating passable landscapes.

If you would like to experiment with training your own GAN you can take a look at the python notebook I developed for this project here. The notebook makes it relatively easy to get started with sections for pre-processing your training data, training the GAN, and performing some cleanup. It runs on Google Colab so you don't even have to install anything! Some further examples of landscapes generated by my GAN are shown below. Note that these took many days of training to achieve so don't be discouraged if the results at the beginning seem lackluster.

More resources on GANs:

Beginner's Guide to GANs

Wikipedia: GANs

How AI Marketplace Works

AI Marketplace contains two different interfaces. The first one consists of a web portal for human users to upload and manage their AIs. This allows people to specify the AI's name, its code as a zipped folder, and an optional surcharge that will be added onto the fee charged by AI Marketplace for each piece of commissioned art. AI owners can also delete an AI if they no longer want to offer it's services.

The other interface is for machines such as IOTA Canvas to communicate with the Marketplace and request and pay for services. It exposes an API that allows internet-connected devices to request the list of AI artists maintained by the Marketplace, commission a new artwork, and download the artwork.

The general flow of functionality for AI Marketplace follows these general steps:

1. A human uploads a new AI as a zip file that contains all the necessary files to run the AI and specifies the surcharge they would like to receive for each commissioned artwork.

2. A device requests a new piece of art, so AI Marketplace generates a new IOTA address and sends it to the requesting device along with a secret key. The requested payment is based on the AI surcharge and how long the AI takes to run, which will be proportional to the amount of resources used to power the AI. An example of the AI Marketplace's response to the request for new art is below. The "status_addr" is the address that can be used to check on the status of the art generation once payment has occurred. The "retrieve_addr" is the address that can be used to download the art once it has been created (see step 5).

{
  "iota_addr": "J9ELTXQMVQTZUCVHJNKLXUUKQDRSMENJYLCRVIAJCYSIBRBP9HTLKHAMO9SRTFVZYRNSNIKLMOFJWVRYZ", 
  "job_id": 58, 
  "key": "81244f70c40443c4aaf47b3db2aa2810", 
  "retrieve_addr": "/58/retrieve-art", 
  "status_addr": "/58/status"
}

3. AI Marketplace monitors the IOTA address for up to ten minutes by repeatedly checking the balance of the address to see if it is equal to the required payment.

4. Once the required payment is received, AI Marketplace starts the requested AI in a sandbox by calling a standardized run() function which takes as input the place to store the generated artwork.

5. When the device that commissioned the artwork requests to download the new art and provides the correct secret key, AI Marketplace serves the appropriate file.

The proof-of-concept I created for AI Marketplace does most of the above, but does not run the AI in a sandbox. It handles IOTA address generation and balance checking similar to IOTA Canvas. See the AI Marketplace repository for information on how to run the software.

Final Thoughts

This was a great project to begin learning about IOTA because it was interesting and fun. For the most part, working with IOTA is actually quite easy. The APIs provided by the IOTA Foundation and Community make it simple to generate IOTA addresses, check balances, and send IOTA. I actually spent more time working through issues with cross-thread communication than I did with configuring IOTA.

Overall this has been a fun and rewarding project. Check out the repositories and build it for yourself!