Preamble
This article, the 11th in a series on home automation documents how to create and integrate an IoT Desktop Device into an existing home automation system including all the necessary software functionality to enable the successful deployment within a domestic environment.
Picture 1 shows the completed IoT desktop device and Picture 2 shows all the component parts used in the prototype which were 'shoe horned' into the finished article.
Introduction
As mentioned above this Instructable details how to make an IoT Desktop Device whose primary purpose is to present the user with an 'at a glance' summary of temperature and humidity levels published by distributed sensors within an IoT network accompanied by local barometric pressure along with time and date.
The design seamlessly hooks into the MQTT/OpenHAB based IoT network detailed in this series on home automation (HA), building on reused code taken from here. Alternatively as the MQTT interface is fully documented throughout it can be simply be adapted to an existing HA system via MQTT publications/subscriptions.
At it's heart is an ESP8266-12E which is responsible for MQTT communications and controlling all functionality excepting LCD display back-lighting control and gesture sensing which use an ATTiny85 and ATMega328P respectively. The device is fully configurable via text files stored on a local SD card, though calibration and network security parameters can also be programmed via remote MQTT publications.
What parts do I need?
See the bill of materials attached below
What software do I need?
What tools do I need?
What skills do I need?
Topics covered
Series Links
To Part 10 : IR REMOTE CONTROL VIA IOT. PART 10 IOT, HOME AUTOMATION
To Part 12 : Retro Speech Synthesis Part 12 IoT, Home-Automation
This section describes the functionality of the IoT Desktop Device in detail from a users perspective. The video above shows the system in use and demonstrates the various functionality in action.
Pictures 1 through 3 name the main component parts of the IoT Desktop Device
From left to right.
Logging is made to the integral SD card in csv format. The name of the logging file is derived by concatenating the Day=DD, Month=MM, Hour=hh,Minute=mm of the local real time clock. ie DDMMhhmm.csv.
The following local parameters are logged; Date, Time, Temp Local, Humidity Local, Barometric Pressure, Ambient Light Level.
Configuration of these buttons is achieved via editing a text file named 'conbutin.txt'. This file is resident on the integral SD card and must contain 6 entries as shown below;
Note : The entries are in the form of MQTT Topics and their respective payloads. If an entry is not required then the key word 'null' must be placed in both Topic and Payload positions
Located on the top of the enclosure in the middle of the buttons, this device monitors local ambient light levels and publishes them via the topic 'WFD/THBSen/7/ALSStatus/1'. Units are in Lux.
The ambient light sensor is also used to trigger a change in System LCD illumination. As the local light levels increase above a user definable threshold the brightness of the display will also increase to give better contrast.
The converse is also true for decreasing light levels.
Configuration of these parameters is achieved via editing a text file named 'confvals.txt'. This file is resident on the integral SD card and must contain 7 entries as shown below;
Note : Entries 6 and 7 are general configuration items used to set the delay before an enumerated sensor value will scroll and to enable gesture control respectively.
The IoT Desktop Device is also fitted with a simple gesture detection sensor on the front of the enclosure which is capable of reconising the following; Left, Right, Up, Down and Near.
Configuration of these parameters is achieved via editing a text file named 'congesud.txt'. This file is resident on the integral SD card and must contain 4 entries such as shown below by way of example;
This display presents the user with 9 items of information as shown in picture 4 above;
Logfile name
When not logging the display shows the words 'NO LOGGING'. Once logging has been initiated the display generates file name as in the description for the 2nd button and starts logging local ambient values to a csv file on the SD card.
Connectivity Icons
These icons indicate the various states of IoT network connection. See picture 5 above. Here you can observe there are 5 icons ascociated with 3 connection states.
AP mode : If the IoT Desktop device fails to connect to your WiFi network it will become and access point on it's own network. By doing so it will allow you to redefine the parameters necessary to connect.
MQTT Broker : This shows two states, connected and not connected. Helpful in determining if there is an issue with the MQTT Broker on your IoT network
WiFi Network : This also shows two states, connected and not connected.
System time
This is the time as read from the on-board real-time clock presented in 24hr format
Barometric pressure
This is the local barometric pressure reading as taken from the internal BMP085 barometric pressure sensor in millibars (mB). The value of barometric pressure is published by the device via the topic 'WFD/THBSen/7/BarometricStatus/1'.
Barometric trend
This gives a simple indication of the barometric trend; rising, stable or falling. See picture 5 above for the actual icons.
System date
This is the date as read from the on-board battery backed real-time clock.
Enumerated sensor name
This is the user assigned name for a given temperature or humidity subscribed topic. Here the user makes the association with say, a temperature and the Hallway, or humidity and Master Bedroom. In each case, once the display cycles through it's list of subscribed temperature and humidity topics, the enumerated sensor name is also displayed via association.
By doing so it makes the sensor information more readable to the user. A typical entry in the 'consenin.txt' configuration file would be;
Temperature
The temperature of the associated subscribed topic paired with an enumerated sensor name. In degrees Celsius.
Humidity
The humidity of the associated subscribed topic paired with an enumerated sensor name. In %age of relative humidity.
This holds all configuration data and web pages when in AP mode. Configurations files are as follows;
These are controlled remotely and can be used as general indication.
Led 1 : This led is controlled by publishing a payload of '0' or '1' to the following topic 'WFD/THBSen/7/Led/Command/1'
Led 0 : This led is controlled by publishing a payload of '0' or '1' to the following topic 'WFD/THBSen/7/Led/Command/2'
Led 0 has special functionality. During the initialisation phase at start up, the led flashes to indicate the state of the connection or the failure code if a connection can't be made.
Note : Local Temperature, Humidity and Heat Index values are published via the following topics respectively;
A full circuit diagram of all the electronics is given in picture 1 above, along with a detailed PDF below.
The desktop device contains three processors;
IC4 : ESP8266-12E
As mentioned above this processor is responsible for controlling the main functionality of the desktop device along with the WiFi back-link into the network and onward connection to the MQTT Broker.
This processor is directly connected to the integral SD card reader and externally mounted DHT22 temperature/humidity sensor for ease and speed of operation. As I/O on the ESP8266 is limited, extra I/O is provisioned via the on-board I2C interface on GPIO Pins 5 (14) SCL and 4 (13) SDA.
This I2C bus connects the following devices;
U1 : ATTiny85
This processor forms an I2C interface (address 0x08) to the system LCD which allows for gradual fading of the back-light and is based on the two following instructables, here and here. I non-destructively reverse engineered the PCF8574A Serial to Parallel I2C Module design and discovered it was possible to control LCD back-light intensity by connecting directly to the two LED pins on the module (marked here as X-Y connector in picture 2 above, or J3 in pic 3). Transistors T1, T2 and resistors R18, R19 form the control circuitry which enable PWM of the LCD back-light.
Given smooth fading with easing functions is processor intensive I ported the control of fading to an ATtiny85.
U3 : ATMega328P
This combination of generic I/O IC6 connected to U3 forms the non-blocking gesture detection system (see here for further details of implementation) and is 3v3 compatible. Connection to the APDS9960 is via J14 which is supplied with 5v, but is 3v3 compatible via an on-board 3v3 regulator. This device also has an I2C adress of 0x39, though this is not exposed to the ESP8266, being directly connected to U3. Picture 4 above shows a logic analyser plot of the gesture sensor in action.
The capability to reset the gesture sensor electronics was included in hardware (R37, R42, R36, D2, C18 and T4), but has not been implemented in software as the subsystem has remained reliable since operation over some months.
Mixed supply rail interface (3v3, 5v)
FETs Q1 and Q2 provide the 3v3 - 5v interface for the I2C communications. Pull up resistors ROpt1...4 were included just in case, but not used in the final design as the modules provisioned sufficient coupling to the respective supplies.
In-circuit programming
The electronics was designed to allow for the in-circuit re-programming of the ESP8266 and ATmega328P for easier debugging;
Mixed supply rails
The desktop device requires the use of both 5v and 3v3 supply rails.
Supply to the device is first conditioned via an SMS buck converter which efficiently drops a wide range of supply input (9v - 15v) to 7.6v, decoupled via C2, this is fed directly to;
So far all of my electronics Instructables have been prototyped on solderless breadboard as proof of concept and constructed on veroboard in final design.
It's always been my intention to commit my designs to PCB but the block to this was lack of good CAD tools (by good I mean cheap) and the prohibitively expensive cost of PCB manufacture in the UK.
I had initially purchased a perpetual 'EAGLE HOBBYIST' user licence for around £100 with the intention to use this toolchain for both schematic capture and PCB design. Things were looking good, I noticed Adafruit and Farnell were creating footprints for all their parts so in theory it would be relatively easy to put together a design sourcing the components from Farnell and using a local (but expensive) PCB manufacture house I'd located.
Before writing this Instructable, all my schematics were done with Eagles, though I hadn't as yet gone so far as to create a PCB partly down to not fully understanding the design flow and there not being a vast amount of training collateral available. I did go so far as to purchase Mitchell Duncan's 'Learning to fly with Eagle' (despite it taking forever to get published) and Simon Monk's 'Make your own PCBs with Eagle', but to-date they remain unused, in pristine condition.
However, with the recent takeover of CadSoft by Autodesk all this changed, despite all the usual dishonest reassurances big business offer they reneged and moved over to a subscription based model, something I really dislike. Rightly or wrongly my view is when I purchase software I own it in perpetuity. It's up to the software company to come up with a model that fits, or I simply won't purchase your product.
Consequently I was back to square one.
However all this recently changed after I noted a great deal of interest in Kicad. After much 'Googling' I found a plethora of information, User Forums, YouTube videos (specifically the truly outstanding work by Chris Gammell at 'Contextual Electronics') etc. All my questions were answered and designing a PCB was actually quite easy.
I now use Kicad for both my schematic diagrams and PCB design. See Pics 1 and 2 above.
Ok, so now I had a good tool chain, plenty of instruction and training but I still needed to use an expensive UK manufacturer to get the PCB fab'ed.
That is until JLCPCB arrived on the scene...
This company seems to have come from 'left field', offering an insanely cheap price for a dual sided PTH PCB for $2 and is now within easy reach of the hobbyist.
Ok the $2 thing is for your first order and there are some restrictions on size, but still they offer outstanding value for money.
Pics 3...6 show my JLCPCB delivery. The box was a bit worse for wear but the packing protected the contents. As you can see I received a pack of 5 PCBs, double sided, plated through hole with silk screen and green solder mask. The thinnest trace I am using is approx 8mil to get the density I needed (this was my first board and it made routing easy).
I chose the basic board package and followed the instuctions and (video) on how to generate and upload your Kicad Gerber files. Once uploaded I was presented with a 'zoomable' image of my board which gave me a 'warm feeling' everything was ok. It was almost like they wanted my business (strange concept in the UK). I paid with paypal and watched how my boards were progressing through the production cycle via my account login.
Just over a week later my boards arrived through the post and the rest is history.
Pic 7 shows my set up for populating the PCB. I used a microscope as my eyes aren't as good as they used to be, and for the most part I used SMT components which are tiny.
Pics 8 to 10 show the board fully populated and the DIL components inserted.
Finally Pic 11 shows the board fully installed and wired up in it's enclosure.
Board Layout
All connections to the board are made via 0.1" Molex connectors to facilitate assembly/disassembly. The RTC and Barometric sensor are located away from the ESP8266 RF source to minimise the risk of interference. The on-board +5v regulator is soldered to a large thermal relief to give better dissipation under operation. I used oversized pads for the 16MHz Xtal on the ATMega328P to allow me to flow solder to all four corners of this SMT device.
.
Note : A full copy of the PCB and schematic are attached below.
This IoT Desktop device contains six key software components as shown in pic 1 above.
SD Card
This is the external SD SPI Flash Filing System and is used to hold the following information (see pic 2 above);
mDNS Server
This functionality is invoked when the IoT device has failed to connect to your WiFi network as a WiFi station and instead has become a WiFi access point something akin to a domestic WiFi router. In the case of such a router you would typically connect to it by entering the IP Address of something like 192.168.1.1 (usually printed on a label affixed to the box) directly into your browser URL bar whereupon you would receive a login page to enter the username and password to allow you to configure the device. For the ESP8266-12E in AP mode (Access Point mode) the device defaults to the IP address 192.168.4.1, however with the mDNS server running you only have to enter the human friendly name 'DSKTOPSRV.local' into the browser URL bar to see the 'IoT Desktop Device Home Page'.
MQTT Client
The MQTT client provides all the necessary functionality to; connect to your IoT network MQTT broker, subscribe to the topics of your choice and publish payloads to a given topic. In short it provisions IoT core functionality.
HTTP Web Server
As mentioned above, if the IoT device is unable to connect to the WiFi network whose SSID, P/W etc. is defined in the Security Information file held on the SD Card the device will become an Access Point. Once connected to the WiFi network provided by the Access Point, the presence of an HTTP Web Server allows you to directly connect to the device and change it's configuration via the use of an HTTP Web Browser it's purpose being to serve up the 'IoT Desktop Device Home Page' web page which is also held on the SD Card.
WiFi Station
This functionality gives the IoT device the capability to connect to a domestic WiFi network using the parameters in the Security Information file, without this your IoT device will not be able to subscribe/publish to the MQTT Broker.
WiFi Access Point
The ability to become a WiFi Access Point is a means by which the IoT device allows you to connect to it and make configuration changes via a WiFi station and a browser (such as Safari on the Apple iPad). This access point broadcasts an SSID = "DSKTPDEV" + the last 6 digits of the MAC address of the IoT device. The password for this closed network is imaginatively named 'PASSWORD'
Preamble
To successfully compile this source code you will need a local copy of the code and libraries outlined below in Step 12,References Used. If you are not sure how to install an Arduino library go here.
Overview
The software makes use of the state-machine as shown in pic 1 above (full copy of source here). There are 5 main states as outlined below;
The events controlling transitions between states are described in pic 1 above. Transitions between states is also governed by the following SecVals parameters;
As mentioned above if the IoT device is unable to connect as a WiFi Station to the WiFi network who's SSID and P/W is defined in secvals.txt held on the SD Card the IoT device will become an Access Point. Once connected to this access point it will serve up the 'IoT Desktop Device Home Page' as shown above in Pic 2 (by entering either 'DSKTOPSRV.local' or 192.168.4.1 into your browsers URL address bar). This home page allows the reconfiguration of the IoT Desktop Device via an HTTP browser.
Remote Access whilst in the ACTIVE state
Once connected to the MQTT Broker it is also possible to both re-calibrate and reconfigure the device via MQTT topic publications. The file calvals.txt has R/W access and secvals.txt has write only access exposed.
User debug
During the boot sequence the IoT device blue System led 0 gives the following debug feedback
The above visual indication is a legacy of the first implementation of IoT connectivity. In this instance as the IoT desktop device also possess an LCD there are 3 connectivity icons which are also updated and the device goes through its various states of connection (pic 3 above).
IoT Desktop Device Functionality in ACTIVE State
Once in the ACTIVE state the ESP8266 enters a continual loop calling the following functions; timer_update(), checkTemperatureAndHumidity(), checkBarometricPressure(), checkALSValue(), updateTimeDate(), readAPDS9960(boolean *bGestureAvailableFlag, uint8_t *uiGestureValue), handleGesture(), updateDisplay(), checkButton(), updateLogging(). The net result of which has been designed to both present the user with parametric environmental data (which is regularly published) and provision a means of interaction/control with other devices via MQTT publications in an IoT infrastructure.
A comprehensive list of all topic subscriptions and publications including payload values is included in the source code.
Use of linked lists
A linked list is a dynamic data structure, similar to an array, differing in so far as it is created with no elements and dimensioned whilst the programme is running. This means the size of the array does not have to be known when the source code is compiled. A linked list (or more accurately a doubly linked list in this case, 'ptrHeadOfSensorInstances') is used here to allow the addition of enumerated temperature/humidity sensor names and their associated topics without the need to recompile the source code. All the user needs to do is remove power, extract the SD card, edit the 'consenin.txt' file and re-insert. Once the IoT desktop device power supply is reconnected the software will re-read 'consenin.txt' and add the enumeration of interest to the linked list. During normal operation in the ACTIVE state, the software passively listens for temperature and humidity publications associated with these topics. When one is received the corresponding temperature/humidity value is dynamically updated in the linked list. Concurrent with this activity, the software periodically sequentially scans the linked list and updates the system LCD with the enumerated sensor name along with the stored value for temperature and humidity. As the linked list is circular the pointer used to scan the list ('tmpSensorInstancePtr') need only be incremented as incrementing beyond the end of the list will point to the front of the list (ie. the pointer wraps inherently).
Picture 4 illustrates linked lists of various lengths and picture 5 depicts how a new element is added to the end of a linked list (the action at start up).
Simple cooperative RTOS
As mentioned in the introduction, this code also contains a simple implementation of a cooperative RTOS (Non-Preemptive, see pics 6 and 7 above. To achieve this, strictly, none of the code in the loop() function is 'blocking' ie. results in an indefinite execution loop (the implementation for gesture sensing with the APDS9960 was specifically designed with this in mind, see here for further details). Here the functions; timer_update(), checkTemperatureAndHumidity(), checkBarometricPressure(), checkALSValue(), updateTimeDate(), readAPDS9960(boolean *bGestureAvailableFlag, uint8_t *uiGestureValue), handleGesture(), updateDisplay(), checkButton(), updateLogging() are iteratively executed on each successive pass through the loop() giving control back when complete (ie. non-preemptive). The outcome of each repetitive execution is dependent upon state variables or timer call back routines each of which independently changing. In this way the code is effectively handling many pseudo simultaneous tasks.
When the IoT device powers up, as part of the boot sequence a file named 'cavals.txt' is read from the SD Card. The contents of this file are calibration constants as indicated above in pic 1. These calibration constants are used to adjust the readings acquired from the sensor to bring them into line with a reference device. There is one further value which defines a reporting strategy for the device and is described below along with the procedure followed to calibrate the sensors.
Reporting Strategy
This parameter determines how the remote sensor reports any ambient parametric changes local to it. If a value of 0 is selected the remote sensor will publish any change it sees in the temperature, humidity, ambient light levels or barometric pressure each time the respective sensor is read (approx every 10 seconds). Any other value will delay the publication of a change by 1...60 minutes. Modifying this parameter allows for optimisation of MQTT network traffic.
Temperature calibration
To calibrate the temperature sensor I followed the same process as outlined here step 4, again using a simple y=mx+c relationship. I used IoT Temperature, Humidity Sensor #1 as the reference device. Values from the sensor are in degrees celcius.
Humidity Calibration
As I possess no means to accurately record or even control local ambient humidity, to calibrate the sensor I used a similar approach to that above here step 4, again using Sensor #1 as reference. However the above said, I have recently found an excellent article on the web describing how to calibrate humidity sensors. I may well try this approach in the future. Values from the sensor are in %age of relative humidity.
Barometric Pressure Calibration
Again as I possess no means to accurately record let alone control local barometric pressure, whilst I added parameters to calibrate zero and scale, they are not used. Values from the sensor are in millibars.
Barometric Trend
Barometric trend is calculated by taking a windowed (MAX_BAROMETRIC_SAMPLES) rolling average of the barometric value read off the BMP085 sensor. The delta between old and new barometric values is determined and scaled '((fNewBarometricTrendAverage - fOldBarometricTrendAverage)/MAX_BAROMETRIC_DELTA)' this 'smoothed' barometric vaue is used to update a 'iBarometricDeltaCount'. A +ve iBarometricDeltaCount results in an upward trend arrow, a zero iBarometricDeltaCount results in a '-' stable icon and a -ve iBarometricDeltaCount gives a downwards trend arrow. Pictures 2 ... 5 give a graphical depiction of the above.
Ambient Light Sensor
The datasheet for this device indicated sufficient accuracy to not require any calibration and therefore I didn't export any parameters in calvals.txt for this sensor. Values from the sensor are in Lux.
As mentioned above the gesture sensing functionality was an integration of the circuit given in this Instructable, pic 1 shows the circuit integrated into the IoT desktop device.
To ensure this gesture sensing worked I used Sigrok PulseView 0.5.0-git-b732b48 along with a '24MHz 8 Channel USB Logic Analyzer' to capture the sequence shown in pic 2 above.
This sequence shows how the IoT desktop device captures a right moving hand gesture;
As mentioned in an earlier Instructable (here) I settled on the topic naming convention outlined in pic 1 above.
Namely, 'AccessMethod/DeviceType/WhichDevice/Action/SubDevice'
It's not perfect but it does allow for useful filters to be applied to see all sensor outputs for a given parametric topic thus allowing for easy comparison as in pic 2 above with MQTTSpy.
It also supports reasonably extensible logical groupings of functionality within a given IoT device.
.
In implementing these topics in software I used hard coded topic strings with fixed, embedded numerical identifiers for each device as opposed to dynamically generating the topics at run time so as to save on RAM and keep performance high.
.
Note : If you're not sure how to use MQTTSpy see here 'Setting Up an MQTT Broker. Part 2 : IoT, Home Automation'
I modified the OpenHAB configuration given in my earlier IoT Instructables (here, here, here and here) adding in the new functionality pics 1...2, along with barometric trend, second ambient light trend, RSSI (Received Signal Strength Indication) trend in pics 3...5 above.
.
If you wish to use the 'stevequinnhousehold' example all you need to do is download the configuration files from here and drop them into the OpenHAB directories shown above in pic 6.
.
Note : If you're not sure how to use OpenHAB see here 'Setting Up and Configuring OpenHAB. Part 6 : IoT, Home Automation'
Testing was carried out using two methodologies;
The first was using MQTT Spy to exercise all of the available subscribed topics and check the published responses (depicted in pic 1 above). As this was achieved manually it can be time consuming and prone to errors, although manual execution does enable 100% coverage.
MQTTSpy was chosen because it is an excellent tool to hand format a given payload and publish it to any topic with ease. It also displays a clear log which is very useful for debugging (pic 2 above).
The second approach was adopted as the source code became more complex (>4000 lines). Increased complexity means longer manual testing cycles and more complex tests. In order to improve the reliability, determinism and quality of tests, automated testing was used via a python test executive (pic 3).
See Step #10 in this Instructable on how automated testing was introduced.
A full copy of the automated tests used in this Instructable is available here.
A video of the automated test sequence in operation is shown above. The sequence executes the following steps;
Note : If you want to use the above automated testing you will need to use the LiquidCrystal_I2C_PCF8574 Arduino library. As far as I know, this is the only library which allows the user to directly read both DD and CG ram in the LCD memory over I2C.
Overall I think project was a success on many parts.
In general
The layout worked well, giving good access to all connectors, ESP8266 programming buttons and CR2032 battery carrier.
The circuit worked 'out of the box' due in most part to all the prototyping I had carried out along with many measurements to ensure everything fitted in. I'm particularly proud of the fit of the BMP085 module which sits between two Molex connectors and overhangs IC6 a PCF8574A I/O expander. Though I think with hindsight I should have integrated the barometric sensor directly on the board rather than use a module.
One thing to note when combining many I2C modules is, invariably the Chinese manufacturer will provision the module with 10K/4K7 pull up resistors attached. By combining many of these devices you are effectively paralleling them up and increasing the bus load. You may find your I2C stops working. If this happens, remove some of the pull-ups.
In my design I had fully prototyped everything so I knew there would be no issues. However to make sure I had 'belt and braces' I also added in pads to my level converter to allow me to remove all module based pull ups and add my own. It turned out I didn't need them in the end.
A note on gesture sensing
The gesture sensor is very precious regarding it's unobstructed view on the world. I had to 'clip off' part of the CS Nylon screws to give better clearance. Also make sure you get the same sensor I chose which only has the detector on the underside of the pcb module, this allows you to get the sensor as far into an aperture as possible. Most other sensors I found had surface mount components clustered on the same side as the detector. This is fine for prototyping but precludes embedding in any final design.
Future development
What would I have done differently?
The following sources were used to put this Instructable together;
Source code for the IoT Desktop Device (this contains a copy of everything)
PubSubClient.h
BounceI2C.h
DHT.h
BH1750.h
DS1307RTC.h
LiquidCrystal_I2C_PCF8574.h
Adafruit_BMP085.h
APDS9960_NonBlocking.h
.
Datasheets
APDS9960 Gesture Sensor
BMP085 Barometric Pressure Sensor
BH1750FVI Light Intensity Sensor
DHT22 or AM2302 Humidity and Temperature Sensor
DS1307Z Real Time Clock
NXP PCF8574 I/O Expander
ATTiny85
ATmega328P
LDV1117v33
LD1117S50TR
.
How to calibrate a humidity sensor
Linked lists
How to install an Arduino Library
Real Time Operating System (RTOS)
.
Software
.