For the last few weeks, I played around with controlling my LEDs over the network by mapping a MIDI keyboards dials to each of my color controllers settings.

Now I want to share my results and the code with you as well as some of the things I learned along the way.


Full HD version of the video above:


In this article, I will give an overview of the client software written in python, showcase a plug and play WiFi setup, how to announce and find the device in the network, and the basic principles of how to get an ESP32 to accept and act out some commands.

If you want to know more about the basic setup for the RGBW controller, check out this article.

And if you want to try this out yourself consider using this amazon affiliate link to get the same keyboard I have and help me pay my bills.


1. The Client

1.1 Reading data from the keyboard

To actually have something to send we need to have a client that reads the MIDI data, transforms it, and generates the command for the server.

We'll be reading the incoming data by parsing the incoming USB data and converting it into usable MIDI data using the midi subpackage of pygame.

I will be using pipenv for easier package management:

# Install pipenv
pip install --user --upgrade pipenv
# Install pygame
pipenv install pygame
# Enter virtualenv
pipenv shell

Now we can have the pygame.midi package print us out all midi devices:

import pygame.midi

def get_devices():
    devices = []
    for n in range(pygame.midi.get_count()):
        info = pygame.midi.get_device_info(n)
        if (info[2]): # Check if the device has the input flag
            print (n,info[1]) # (device id, device name)
            devices.append(n)
    return devices

pygame.midi.init()
print(get_devices())

We will then connect to the MIDI device of our choosing by its device id. Listening in, we will ignore any key-presses and only work with the dials and button presses:

buttons = [*range(16, 23 + 1), 64]
dials = [*range(1, 8 + 1)]

def readInput(input_device):
    while True:
        if input_device.poll():
            event = input_device.read(1)[0][0] # Strip away irrelevant data
            if (event[0] != 176): # Check if this is a change event for a mode/control (this is device channel 1 only, you might want to check 176 to 191)
               	if (debug or event[0] not in [144,]):
                    print (event)
                continue
            if (event[1] in dials): # Check if it's a dial
                dial_handler(event)
            elif (event[1] in buttons): # Check if it's a button
                button_handler(event)
            else:
                print (f"ID: {event[1]}, State: {round(event[2]/127,2)}")
        send_data()


my_input = pygame.midi.Input(device_id)
try:
    readInput(my_input)
except KeyboardInterrupt:
    my_input.close()

Let's take a look at the handler for the dials:

dialdata = {}

def dial_handler(event):
    pos = event[2] / 127 # Remapping to 0 to 1 
    print(f"Dial {event[1]} at position {round(pos, 10)}") # Mandatory print statement
    dialdata[event[1]] = pos # Take a guess.. event[1] is the dial id

Skipping the button code as it is only relevant for switching the color mode which is irrelevant for this article. Also removed some code that uses the second row of the dials for fine tuning purposes, for the same reason.

1.2 Compiling data to be send

The send_data function will now take the dialdata, transform it, and finally send it to the server:

stime = 0
last_fill = [-1, -1, -1, -1]

def send_data():
    global stime # I know I know global vars are ugly but this is not clean code 101
    global last_fill
    ptime = int(round(time.time() * 1000)) - stime
    if (cm and ptime > 10): # cm and ptime will be explained below
        fill = [-1, -1, -1, -1] # -1 represents a "keep value" state
        for i in range(1, 4 + 1): # Just filling the array with our dialdata
            fdata = dialdata.get(i, -1)
            if (fdata == -1):
                continue
            # I thought here was some fine tuning code somewhere...
            fill[i - 1] = fdata
        if (fill == last_fill): # Why would we send a new command if nothing changed?
            return
        last_fill = fill
        f = fill
        m = "manual" # For simplicity sake, check the source code for more interesting stuff
        bridge.send_command(cm, "set", [m, *f]) # "set" and the m, *f array are just the type of the command and it's respective data.. more on that later
        stime = int(round(time.time() * 1000))

First of all, what is this stime and what is it doing there?
Well, to not overwhelm the ESP by directly sending the status of the MIDI dials to it, causing it to queue up commands that will be overwritten by the newest one anyway,
we will have to limit the number of times we actually send the data.

In my case, I will wait 10 ms before sending the next command.
Which is coincidentally about the same time the ESP takes to update its color and become ready for the next command.

To be able to understand how we are sending data we need to understand what a command is first.

It is an identifier for the action that should be performed, followed by its respective options. It is then sent over a TCP connection to the server.
A command to turn on all LEDs looks like this:

set manual 1 1 1 1 \n
│   │      │ │ │ │
│   │      │ │ │ └> White value
│   │      │ │ └> Blue value
│   │      │ └> Green value
│   │      └> Red value
│   └> how should the parameters be interpreted?
└> set color to ...

Note the newline character on the end as it will be essential for performance later on.

1.3 Sending the data

To construct the command from its parameters, encoding it to be sent as bytes and actually sending it we will use this code:

def send_command(socket, command, options):
    socket.send(f"{command} {' '.join(str(x) for x in options)} \n".encode("utf-8"))

But how do we get this socket?

This socket or cs from the send_data function further up stands for command socket and is basically our command line that gets injected into the send_command function to make it stateless and therefore easier to use.

To open this socket we first have to find out the IP of the ESP.
Thankfully the ESP is constantly screaming into the endless void of the network that it indeed exists. (more on that later) And if we are willing to listen to its desperate screams by opening a specific port that it broadcasts to we can get its IP:

from socket import *

def open_listener():
    bclistener = socket(AF_INET, SOCK_DGRAM) # Some setup stuff
    bclistener.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) # Some more setup stuff
    bclistener.bind(('0.0.0.0', 24444)) # Listening on port 24444
    return bclistener

def await_controller(bclistener):
    while True:
        message = bclistener.recvfrom(64) # Reading message from socket
        if (message[0] == b'ThatLEDController online!'): # Checking for specific message
            return message[1][0] # Returning ip


bc = open_listener()
ip = await_controller(bc)

Now that we got the IP we can plug that into a socket and open our command socket. Here is a handy function for that:

def open_sender(ip):
    cmsender = socket(AF_INET, SOCK_STREAM) # Some setup
    cmsender.connect((ip, 25555)) # Connecting to port 25555 at 'ip'
    return cmsender
    
cs = open_sender(ip)

Now that the client is implemented, let's build the server side, shall we?


2. The Server

2.1 Connecting to the network

The basic way to connect an ESP32 to the network would be to include the WIFI library, throw some (generally hardcoded) credentials against it and hope it connects:

#include <WiFi.h>
 
const char* ssid = "SSID";
const char* password =  "Password";
 
void setup() {
  Serial.begin(115200);
 
  WiFi.begin(ssid, password);
 
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.println("Connecting to WiFi..");
  }
 
  Serial.println("Connected to the WiFi network");
 
}

But luckily there is a library called WiFiManager that does this and much more for us with little to no configuration. It tries to connect to a predefined or saved network and when that fails, opens an AP on its own that that allows you to specify a network that it will then try to connect to:

#include <WiFiManager.h>
#include <WiFi.h>
#include <WebServer.h>

WiFiManager wifiManager;

void setup() {
  // Configure wifi
  WiFi.mode(WIFI_STA);
  
  // Set dark mode
  wifiManager.setClass("invert");
  
  // Set manager to not blocking
  wifiManager.setConfigPortalBlocking(false);
  
  // Automatically connect to saved network
  // or open network named "ThatLEDController"
  // You can also pass this function a predefined network
  wifiManager.autoConnect("ThatLEDController"); 
}

void loop() {
  // Non blocking process function
  wifiManager.process();
}

OK, WiFi connection check, now how about allowing the client to actually find us?


2.2 Finding the ESP

To reliably connect to the device without relying on a hardcoded IP address, we will let the ESP tell it to us on its own. Well, not just us but rather the entire network.  

We will use something called broadcasting for that, that allows you to send some data to a specific IP address in the network(most of the time something with 255 on the end) and it will be relayed to every client connected:

#include <AsyncUDP.h>
#include <WiFi.h>

// Non blocking wait helper
#define runEvery(t) for (static uint16_t _lasttime;\
                         (uint16_t)((uint16_t)millis() - _lasttime) >= (t);\
                         _lasttime += (t))

AsyncUDP udp;

void setup() {
  // Setup wifi
}

void loop() {
  runEvery(2000) { // Please don't spam the network
    if (WiFi.status() == WL_CONNECTED) { // Check if we are connected
      IPAddress broadcastip;
      broadcastip = ~WiFi.subnetMask() | WiFi.gatewayIP(); // Get broadcast address for current address range
      if(udp.connect(broadcastip, 24444)) { // Open connection to port 24444
        udp.print("ThatLEDController online!"); // Send message
      }
      udp.close(); // Close connection again
    }
  }
}

Moving on to the actual command server.

2.3 Setting up a basic server

Here is the basic setup for a simple single client TCP server running on port 25555:

#include <WiFi.h>

WiFiServer wifiServer(25555); // Open server on port 25555
WifiClient wifiClient;

void setup() {
  // Setup wifi
  
  wifiServer.begin(); // Start server
}

void CheckForConnection() {
  if (wifiServer.hasClient()) { // Check if there is already a client
    if (wifiClient.connected()) { // Check if the client is actually connected
      wifiServer.available().stop(); // Reject new client
    } else {
      wifiClient = wifiServer.available(); // Accept new client
      Serial.println("Client connected!");
      wifiClient.write("Hey there!"); // Send welcome message
    }
  }
}

void loop() {
  CheckForConnection();
}

You can test that it works using netcat:

nc 0.0.0.0 25555

Now there should be a server running that we can extend to accept some commands.


2.4 Receiving commands

First, we will have to read the command into a variable.
We will do this with a blocking function, as we have defined the commands to end with a newline which makes it faster than reading it in chunks. Add this part after the connection check:

if (wifiClient.available()) {
    std::string cm(wifiClient.readStringUntil('\n').c_str());
    Serial.println(cm.c_str());
}

We will now check if the first three characters are actually "set" and if that's true pass is on to the handler function:

if (wifiClient.available()) {
    std::string cm(wifiClient.readStringUntil('\n').c_str());
    if (cm.rfind("set ", 0) == 0) {
        setCmd(cm);
    }
}

The handler function will then split the command at the spaces and check if it is actually long enough. After that the respective values will be extracted and a new color is set:

std::string mode = "hsv";
std::array<int, 4> fill = {0, 0, 0, 0};
std::array<std::string, 4> modes = {"manual", "rgb", "hsv", "hsi"};

// Helper function to split string at delimiter
std::vector<std::string> split (const std::string& str, char delimiter) {
  std::vector<std::string> tokens;
  std::string token;
  std::istringstream tokenStream(str);
  while(std::getline(tokenStream, token, delimiter)) {
    tokens.push_back(token);
  }
  return tokens;
}

void setCmd(std::string str) {
  std::vector<std::string> tokens = split(str, ' '); // Split string at spaces
  
  if (tokens.size() <= 1) { // Return if the command is to short
    return;
  }
  
  // Get what could be the mode from the extracted tokens
  std::string tmode = tokens[1]; 
  
  // Loop over tokens and extract color, ignore if the data is negative
  // resolution_factor is the pwm range
  int tsize = tokens.size() - 2;
  std::array<int, 4> color;
  for (size_t i = 0; i < 4; i++) {
    color[i] = i < tsize ? atof(tokens[i + 2].c_str()) * config::resolution_factor : fill[i];
    color[i] = color[i] >= 0 ? color[i] : fill[i]; 
  }
  
  // Set mode to extracted mode, if extracted mode is valid
  if (std::find(std::begin(modes), std::end(modes), tmode) != std::end(modes)) {
    mode = tmode;
  }
  
  
  fill = color;
}

And now there be light.


3. Conclusion and a look into the future

For a single device, this setup works great, even though it has some problems with reconnecting to the device. But that can easily be fixed by allowing new devices to override the already connected one.

I also added a preprocessor flag to my light controller that allows to deactivate anything but the command server, network code, and basic pin setup. This should save some resources when the direct interface is not needed.

In the long run, however, I want to expand the system to allow for more devices in the same network and create an interface to control them. (electron-based maybe?)

Furthermore, I might even try to connect the MIDI keyboard directly to the ESP as it technically acts as a USB host. But having to handle the USB to usable MIDI data conversion myself might be a bit overwhelming.

Anyway, that's all, thanks for reading!


Check out my sourcecode for the >client< and >server<