Okay this was also next level for me! I had absolutely no experience with compiling c code to binary files. But that makes it so attractive for me to jump into the deep and see how far I get. I have to admit that it took me a lot of time. Eventually it also succeeded and got everything working. I also have learned a lot! Now the question is, do you take the red pill, then we go further, do you take the blue pill and go for the pre-compiled files for easy installation?
Okay you took the red pill let’s see how deep the rabbit holes goes.
NOTE: See the latest blog ESP HomeKit SDK – Revised Installation — Scheduled on February 2nd 2021.
Installing prerequisites
Before we can install the software we need to install some software. Because I using a MacBook Pro with macOS 10.14.4 (Mojave) at this time of writing. I’m using Docker to install some docker files. You can also use Oracle Virtual Box and install Ubuntu, but this is a little more complicated, because you need to switch between the two OSes. When you install the docker files you can compile your code directly on your Mac.
Download Docker
Okay let’s get started go to https://www.docker.com and press “Get Started” and press on “Download for Mac“. before you can download your free version you need to make an account. be sure to write it down because you will need it later. When you made your account you can login on the site and download your free version.
Docker Desktop for Mac
Docker Desktop for Mac is an easy-to-install desktop app for building, debugging, and testing Dockerized apps on a Mac. Docker Desktop for Mac is a complete development environment deeply integrated with the Mac OS Hypervisor framework, networking, and filesystem. Docker Desktop – Mac is the fastest and most reliable way to run Docker on a Mac.
Installation
System requirements
Docker Desktop – Mac works on OS X Sierra 10.12 and newer macOS releases.
Install it
Double-click Docker.dmg to start the install process. When the installation completes and Docker starts, the whale in the top status bar shows that Docker is running, and accessible from a terminal. when you click on the whale you can login with your credentials that you used to download you free copy.
Now that we have installed this part we can install some docker containers.
Atom
Atom is a free and open-source text and source code editor for macOS, Linux, and Microsoft Windows with support for plug-ins written in Node.js, and embedded Git Control, developed by GitHub. Atom is a desktop application built using web technologies. Most of the extending packages have free software licenses and are community-built and maintained. You can download your free copy here.
Containers
Create an empty directory and change into it. Create a file in Atom esp-sdk-dockerfile with following content:
FROM ubuntu:16.04 as builder
RUN groupadd -g 1000 docker && useradd docker -u 1000 -g 1000 -s /bin/bash --no-create-home
RUN mkdir /build && chown docker:docker /build
RUN apt-get update && apt-get install -y \
make unrar-free autoconf automake libtool gcc g++ gperf \
flex bison texinfo gawk ncurses-dev libexpat-dev python-dev python python-serial \
sed git unzip bash help2man wget bzip2 libtool-bin
RUN su docker -c " \
git clone --recursive https://github.com/pfalcon/esp-open-sdk.git /build/esp-open-sdk ; \
cd /build/esp-open-sdk ; \
make STANDALONE=n ; \
"
FROM ubuntu:16.04
RUN apt-get update && apt-get install -y make python python-serial
COPY --from=builder /build/esp-open-sdk/xtensa-lx106-elf /opt/xtensa-lx106-elf
ENV PATH /opt/xtensa-lx106-elf/bin:$PATH
Create a file Create a file in Atom esp-rtos-dockerfile with following content:
FROM ubuntu:16.04 as builder RUN apt-get update && apt-get install -y git RUN git clone --recursive https://github.com/Superhouse/esp-open-rtos.git /opt/esp-open-rtos FROM esp-sdk:latest COPY --from=builder /opt/esp-open-rtos /opt/esp-open-rtos ENV SDK_PATH /opt/esp-open-rtos
You can also download these files from my GitHub here for esp-sdk-dockerfile and here for esp-rtos-dockerfile
Open the Terminal app. Click the Finder icon in your dock. Click Go. Click Utilities. Double-click Terminal. To build esp-sdk Docker container, run this command:
docker build . -f esp-sdk-dockerfile -t esp-sdk
Open the Terminal app. Click the Finder icon in your dock. Click Go. Click Utilities. Double-click Terminal. To build esp-rtos Docker container, run this command:
docker build . -f esp-rtos-dockerfile -t esp-rtos
Make a new folder called ESP. Where Your_Username is you Mac Username, Run this command:
mkdir /Users/Your_Username/ESP
Change into the directory by typing:
cd ESP
Now you can clone esp-open-rtos repository within this directory
git clone --recursive https://github.com/SuperHouse/esp-open-rtos.git
Now we only have to setup the enviroment variables:
export SDK_PATH=`pwd`/esp/esp-open-rtos
Install esptool
Esptool is a Python-based, open source, platform independent, utility to communicate with the ROM bootloader in Espressif ESP8266 & ESP32 chips.
pip install esptool
we also have to clone esp-homekit-demo repository because this is the main repository:
git clone --recursive https://github.com/maximkulkin/esp-homekit-demo.git
Configuring esp-homekit-demo
Copy wifi.h.sample and save it as wifi.h.
cd esp-homekit-demo
Open the nano editor.
nano wifi.h.sample
Edit it with correct WiFi SSID and PASSWORD. Press CTRL + O, change the name to Wifi.h followed by Enter. Then press CTRL + X.
Note: The Wifi.h.sample part is not necessary when you use LifeCycle Manager.You can leave the SSID and PASSWORD blank. Just follow these steps for good practice.
To build an example, first change into esp-homekit-demo directory (into it’s root directory:
cd esp-homekit-demo
Compiling the code
Now its time to compile the code by default it is done by typing:
docker run -it --rm -v `pwd`:/project -w /project esp-rtos make -C examples/led all
If you use ESP8266 with 4MB of flash (32m bit), then you’re fine. If you have 1MB chip, you need to add the following:
FLASH_SIZE=8
Because I use LifeCycle Manger (more about this later) to distribute my Binary files you have to change the default address 0x7a000 to 0x8c000 so you need to add the following:
HOMEKIT_SPI_FLASH_BASE_ADDR=0x8c000
Then build example you want (e.g. Led) by running
docker run -it --rm -v `pwd`:/project -w /project esp-rtos make -C examples/led FLASH_SIZE=8 HOMEKIT_SPI_FLASH_BASE_ADDR=0x8c000 all
Wait until the compiling is done.
Now when you change to the examples directory
cd examples
and the to the led directory
cd led
you will see two new folders called Build and Firmware open the Firmware directory.
cd Firmware
Here you wil see the Led.bin file. Theoretically you are finished now. you can Upload the bin File you Your Esp and it should work.
But of course I don’t stop here, let’s go further with a tool that makes our life a little bit easier.
INstalling the Firmware
Before we go to the part where we actually upload the firmware to your ESP we first need to get some other software. The Software we need is the Open Source FreeRTOS-based ESP8266 Software Framework, or sort ESP Open RTos. Furthermore we need a distribution system in this case we use GitHub as our online distribution system. And a smart manger that can install our Firmware, herefore we use LifeCycle Manager. We also need a good code editor like Atom.
ESP Open RTos
ESP Open RTos is a community developed open source FreeRTOS-based framework for ESP8266 WiFi-enabled microcontrollers. Intended for use in both commercial and open source projects. Originally based on, but substantially different from, the Espressif IOT RTOS SDK.
You need two files to install the ESP Open RTos. You can download them here:
File name: rboot.bin
Version: 1.4.2
File name: blank_config.bin
Version: 1.4.2
LifeCycle Manger
LifeCycle manger will Initial install, configure your WiFi settings and preform over the air firmware upgrades for any esp-open-rtos repository on GitHub. This is a program that allows any simple repository based on esp-open-rtos on esp8266 to solve its life cycle tasks. You can find more information about LifeCycle Manager here. You can download the binary file down here from my GitHub.
File name: OTABoot.bin
Version: 2.0.2
Update 20-03-2020: read https://www.studiopieters.nl/esp8266-life-cycle-manager-trouble and then https://www.studiopieters.nl/esp8266-life-cycle-manager-trouble-update
Atom
Atom is a free and open-source text and source code editor for macOS, Linux, and Microsoft Windows with support for plug-ins written in Node.js, and embedded Git Control, developed by GitHub. Atom is a desktop application built using web technologies. Most of the extending packages have free software licenses and are community-built and maintained. You can download your free copy here.
PREPARING you code for LifeCycle manager
At first we have to change a few lines of code, why you ask? Do you remember the Copy wifi.h.sample and save it as wifi.h. step? This step is necessary for your ESP to connect to your network. Only we don’t want to hardcode it into our bin file. Therefor we use Lifecycle Manager. So we need to change a few lines.
Open the Terminal app. Click the Finder icon in your dock. Click Go. Click Utilities. Double-click Terminal. then type:
cd esp
change into folder esp-homekit-demo
cd esp-homekit-demo
then change into the examples folder.
cd examples
Then open a accecory (e.g. Led)
cd led
Now you will see two files:
led.c
makefile
Open Led.c with Atom and change the following lines:
#include <stdio.h>
#include <espressif/esp_wifi.h>
#include <espressif/esp_sta.h>
#include <esp/uart.h>
#include <esp8266.h>
#include <FreeRTOS.h>
#include <task.h>
#include <homekit/homekit.h>
#include <homekit/characteristics.h>
#include "wifi.h"
static void wifi_init() {
struct sdk_station_config wifi_config = {
.ssid = WIFI_SSID,
.password = WIFI_PASSWORD,
};
sdk_wifi_set_opmode(STATION_MODE);
sdk_wifi_station_set_config(&wifi_config);
sdk_wifi_station_connect();
}
to
#include <stdio.h>
//#include <espressif/esp_wifi.h>
//#include <espressif/esp_sta.h>
#include <esp/uart.h>
#include <esp8266.h
> #include <FreeRTOS.h>
#include <task.h>
#include <homekit/homekit.h>
#include <homekit/characteristics.h>
//#include "wifi.h"
#include <wifi_config.h> //Important - add this line!
//static void wifi_init() {
// struct sdk_station_config wifi_config = {
// .ssid = WIFI_SSID,
// .password = WIFI_PASSWORD,
// };
// sdk_wifi_set_opmode(STATION_MODE);
// sdk_wifi_station_set_config(&wifi_config);
// sdk_wifi_station_connect();
//}
then we have to adjust this line:
wifi_init();
homekit_server_init(&config);
to
// wifi_init();
homekit_server_init(&config);
and place this:
void on_wifi_ready() {
}
Before this line:
void user_init(void)
That’s It now we are done with your led.c code. Now we have to change the make file. Open makefile in Atom and change the following lines:
PROGRAM = led EXTRA_COMPONENTS = \ extras/http-parser \ $(abspath ../../components/common/wolfssl) \ $(abspath ../../components/common/cJSON) \ $(abspath ../../components/common/homekit) FLASH_SIZE ?= 32 EXTRA_CFLAGS += -I../.. -DHOMEKIT_SHORT_APPLE_UUIDS include $(SDK_PATH)/common.mk monitor: $(FILTEROUTPUT) --port $(ESPPORT) --baud 115200 --elf $(PROGRAM_OUT)
PROGRAM = led EXTRA_COMPONENTS = \ extras/http-parser \ extras/dhcpserver \ $(abspath ../../components/esp-8266/wifi_config) \ $(abspath ../../components/common/wolfssl) \ $(abspath ../../components/common/cJSON) \ $(abspath ../../components/common/homekit) FLASH_SIZE ?= 32 EXTRA_CFLAGS += -I../.. -DHOMEKIT_SHORT_APPLE_UUIDS include $(SDK_PATH)/common.mk monitor: $(FILTEROUTPUT) --port $(ESPPORT) --baud 115200 --elf $(PROGRAM_OUT)
UPDATE 15-01-2020: Due to changes in the main repository $(abspath ../../components/esp-8266/wifi_config) \ has to be changes to $(abspath ../../components/esp8266-open-rtos/wifi_config) \ and $(abspath ../../components/common/cJSON) \ to $(abspath ../../components/esp8266-open-rtos/cJSON) \
docker run -it --rm -v `pwd`:/project -w /project esp-rtos make -C examples/led FLASH_SIZE=8 HOMEKIT_SPI_FLASH_BASE_ADDR=0x8c000 all
Wait until the compiling is done.
Open the Terminal app. Click the Finder icon in your dock. Click Go. Click Utilities. Double-click Terminal. Now change to the newly made firmware directory by typing:
cd esp
change into the esp-homkit-demo directory:
cd esp-homekit-demo
then type:
cd examples
and the to the led directory
cd led
you will see two new folders called Build and Firmware open the Firmware directory.
cd Firmware
Once in the Firmware directory you will see a file called led.bin. This is the final bin file we are going to use. Before we are going upload it to our ESP we have to add some security measures with OpensSSL.
OpenSSL is a robust, commercial-grade, and full-featured toolkit for the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols. This is a perquisite by LifeCycle Manager to be sure there is no tampering with you files. So basically the sig files are only to verify/confirm .bin file identity. For more information read more here.
Type the following to create this bin file:
openssl sha384 -binary -out firmware/led.bin.sig firmware/led.bin
and to write/make the actual file:
printf "%08x" `cat firmware/led.bin | wc -c`| xxd -r -p >>firmware/led.bin.sig
Now you have created a file called led.bin.sig. Next step is to make a GitHub repository.
GitHub
What Is GitHub, and What Is It Used For?
GitHub is a website and service that we hear geeks rave about all the time, yet a lot of people don’t really understand what it does. Want to know what all the GitHub hubbub is about? Read on to find out.
To understand GitHub, you must first have an understanding of Git. Git is an open-source version control system that was started by Linus Trovalds—the same person who created Linux. Git is similar to other version control systems—Subversion, CVS, and Mercurial to name a few.
So, Git is a version control system, but what does that mean? When developers create something (an app, for example), they make constant changes to the code, releasing new versions up to and after the first official (non-beta) release.
Version control systems keep these revisions straight, storing the modifications in a central repository. This allows developers to easily collaborate, as they can download a new version of the software, make changes, and upload the newest revision. Every developer can see these new changes, download them, and contribute.
Similarly, people who have nothing to do with the development of a project can still download the files and use them. Most Linux users should be familiar with this process, as using Git, Subversion, or some other similar method is pretty common for downloading needed files—especially in preparation for compiling a program from source code (a rather common practice for Linux geeks).
Git is the preferred version control system of most developers, since it has multiple advantages over the other systems available. It stores file changes more efficiently and ensures file integrity better. If you’re interested in knowing the details, the Git Basics page has a thorough explanation on how Git works.
So eventually you have to make a repository to distribute you firmware.
Repository
A repository (usually abbreviated to “repo”) is a location where all the files for a particular project are stored. Each project has its own repo, and you can access it with a unique URL.
In a nutshell go to GitHub.com and make a account create a new Repository and give it a name (e.g Led). file in the Description (e.g. firmware for esp HomeKit led) and place a tick at “Initialize this repository with a README“. select gitnore and select C and select your licence type (e.g MIT Licence). Now click on Create Repository.
Now click on Releases and the Create a new release. Now you have to fill in the version number (e.g. V0.0.1).
Note: the version number always has to be in x.y.z format
Give your realise a name (e.g LED) and fill in a description if you want to. Then upload the Led.Bin and Led.Bin.Sig file to you new release, and click on Publish release.
Writing the ESP8266
Flashing a program to the ESP8266 is a bit more annoying than flashing an Arduino. When flashing the arduino, all you have to do is press the reset button and release while you upload a program (or even not doing anything if you have FTDI such as in arduino UNO,NANO, MICRO, etc.) and the arduino will start uploading. With the ESP8266 you have to reset the micro-controller and start it in flashing mode using the GPIO0-to-Ground.
ESP8266-01 Scheme
ESP8266-12F Scheme
The sketch shows the ESP8266 connected to FTDI Programmer with a voltage regulator onboard.
NOTE: Be Sure that you have the Jumper placed on the 3.3V setting and not the 5V setting because this will destroy your ESP module!
As you can see two buttons have been added. The Left button, when pressed, connects the RESET pin to the ground and when it is released, connects the RESET pin to the VCC through a pull-up resistor.
The Right button, when pressed, connects the PROGRAM pin to the ground and when it is released, connects the PROGRAM pin to the VCC through a pull-up resistor.
I also added A LED when the ESP is in PROGRAMMING STATE state the LED is ON. When The ESP is in “NORMAL” state the LED is OFF. Using this two buttons you can do all the tasks you need with the ESP8266:
- Working on NORMAL mode – Both buttons are released.
- RESETTING the ESP8266 – Press the RESET button and release.
- Start in FLASH MODE – Press both buttons, release the RESET button and then release the PROGRAM button.
Putting Device Into Flash Mode
To enable ESP8266 firmware flashing GPIO0 pin must be pulled low before the device is reset. Conversely, for a normal boot, GPIO0 must be pulled high or floating. Start in FLASH MODE – Press both buttons, release the RESET button and then release the PROGRAM button.
Go to the directory you made where you put the previously downloaded Root.bin, Blank_config.bin and Otaboot.bin files.
Use esptool.py to flash it in your device. First, erase flash memory from your ESP module:
esptool.py erase_flash
Normally, your ESPPort will be something like (e.g /dev/cu.usbserial-A50285BI). Then, set your device in flash-mode again, and flash the new firmware:
esptool.py -p /dev/cu.usbserial-A50285BI --baud 115200 write_flash -fs 1MB -fm dout -ff 40m 0x0 rboot.bin 0x1000 blank_config.bin 0x2000 otaboot.bin
Note: If you use an old version of esptool, you must change -fs 1MB to -fs 8m.
You must configure wifi network and OTA repository. To configure wifi settings, device generates its own Wifi in AP mode. You must connect to it in order to setup your wifi network. Simply take your iOS device, go to Setting -> Wi-Fi, and search a SSID with LCM- followed of last MAC address, connect to it, and wait a few seconds until a web appears showing you all wifi networks that the device has found. Select yours, and enter password. Don’t touch Join button yet!!
Now, you must configure OTA repository as well. It’s very important that you configure it right, because you can not change it in the future (If you make a mistake, you must erase and flash device again).
OTA repository ( this is your previously made GitHub name followed by your repository name):
Your_Name/led
OTA binary file:
led.bin
To finish initial setup, click Join button and wait about 7 minutes until process finish (While installation is working, device doesn’t show anything, and buttons don’t work). After that, LED turns on for a couple of seconds and you will be able to add your accessory to your HomeKit ecosystem using Home App. LCM will install your Homekt device on your ESP.
TIP: Open your Arduino IDE and then the serial monitor. now you can see the whole installation process!
HomeKit setup
Now it’s time to add your HomeKit accessory.
Open the Home app on your iPhone. Tap Add Accessory. (If you already have other HomeKit accessories paired, tap + in the top-right corner to add an accessory.)
Select Don’t Have a Code or Can’t Scan?
You’ll see a list ( If you are to quick it will not show, just press enter code…) of all available devices currently on your network. Select StudioPieters LED-XXX from the list.
Enter the 8-digit homekit setup code:
111-11-111
compiling and setting up
The test
It takes about 30 seconds for HomeKit to make a connection. After setting up your device whe are ready to test ! Now simply press the reset button in your to turn on and off your LED light. Here in the little movie where I test my setup.
So there it is the whole compiling and installation! This was even hard for me because there are some next level skills and knowledge needed to do this. Thats why in my articles I work with prebuilt binary files, so that also people with less skills can enjoy this project.
Future
I’am still working on this project, like there will be a QR code witch you can scan to add you device. So keep reading my blogs!
Note: To produce and sell HomeKit compatible accessories, your company need to be certified for that (https://developer.apple.com/homekit/, If you’re interested in developing or manufacturing a HomeKit accessory that will be distributed or sold, your company must enroll in the MFi Program.) Espressif have their implementation of HomeKit framework, but it will give you it only if you have MFi certification (notice this text at the bottom of page you mentioned: Please note that the Espressif HomeKit SDK is available to MFi licensees only, and you need to provide the Account Number for verification purposes when requesting the SDK.).This project is a non-commercial implementation of HAP protocol, not meant for commercial use.
REFERENCE
Maxim Kulkin, esp-wifi-config (2019), Library to bootstrap WiFi-enabled accessories WiFi config, https://github.com/maximkulkin/esp-wifi-config Paul Sokolovsky, esp-open-sdk (2019), Free and open (as much as possible) integrated SDK for ESP8266/ESP8285 chips, https://github.com/pfalcon/esp-open-sdk Espressif Systems, esptool (2019), ESP8266 and ESP32 serial bootloader utility, https://github.com/espressif/esptool HomeACcessoryKid, life-cycle-manager (2019), Initial install, WiFi settings and over the air firmware upgrades for any esp-open-rtos repository on GitHub, https://github.com/HomeACcessoryKid/life-cycle-manager