Pages

Showing posts with label raspberry pi. Show all posts
Showing posts with label raspberry pi. Show all posts

Monday, March 27, 2017

Alexa, turn on the water pump - OK

With the major goals achieved for my home automation project I felt it was time to add some bells and whistles. First off, voice activation.



Last November's Black Friday event led to the delivery of an Amazon Echo Dot which has been sitting under utilised in the kitchen. I had seen that the likes of Hive and Tado were offering Skills to control their systems and not wanting to be left behind, I looked into how I could add similar functionality to my own system.

I was thinking along the lines of

  • 'Alexa, boost the kitchen for an hour' 
  • 'Alexa, turn on the hot water pump'
  • 'Alexa, who is calling for heat?'

but after some initial investigation I realised that this would require a custom skill and the building of a voice interaction model. This would be way more effort than the end feature would justify. But all was not lost as Amazon provide a simplified framework for home automation projects, the Smart Home Skill SDK.

Amazon's Smart Home Skill 


This takes care of all the extra work by providing a standardized framework of voice commands which would be adequate for my purposes. Some example utterances would be:
  • Alexa, discover my devices
  • Alexa, set the living room to 20 degrees
  • Alexa, turn on the hot water pump
  • Alexa, set the <device name> to 50%
It looks like there are some 'get' commands too, but only available in the US at present.

I could certainly make use of  'turn on the hot water pump' and decided this would be a good first command to implement.

Getting Started with Smart Home Skills

Amazon provides a step by step guide to creating a Smart Home Skill, starting with a prerequisites list:
Signing up for the developer account and the AWS account were straightforward. The developer account is free and the AWS account has a free tier option for light use (requires a credit card though).

The OAuth part looked less straightforward and I was not sure how this was going to work out. However, I ploughed on through Amazon's step by step guide and discovered that for my purposes I could use Login with Amazon as the OAuth server. This worked fine.

The Amazon guide is very good and provides a dummy lambda function (in python) to get going with. The following steps were enough to get communication from Alexa through to the Lambda function and back.


It is important to take it slowly and follow every step. I missed the relevance of selecting the correct region for the Lambda function (EU Ireland in my case) and ended up with no obvious errors during testing, but with my Lambda function not getting called once I had enabled the skill. I wasted a lot of time getting to the bottom of this mistake.

After ironing out the region issue I was able to successfully ask Alexa to "Discover Devices" and could see the dummy device "Smart Home Virtual Device" appear in the Alexa App. Asking Alexa to "Turn On the Smart Home Virtual Device" resulted in an "OK" from Alexa.

The next steps in the guide are about submitting your skill for certification. These steps are not necessary for personal Alexa skills and can be ignored.

My next step was to replace the dummy lambda function with something that actually communicated with my raspberry pi. This turned out to be remarkably easy.


Adapting the Lambda Function

After completing the step by step guide I had a Lambda function sitting in the AWS cloud which was able to receive messages (directives) from Alexa via my Smart Home Skill. The lambda function was just sample code written in Python and which reported a single dummy device capable of actioning 'Turn On' and 'Turn Off' directives.

The code was a single inline python script with the main entry point 

    def lambda_handler(event, context):

I played around with this code for a while to get a feel for which fields were important and what happened when things weren't set up correctly. All of the directives supported by the Smart Home Skill are listed in The Smart Home Skill API Reference another excellent guide from Amazon.

The Directives I was most interested in were:
  • DiscoverApplianceRequest / Response
  • TurnOnRequest / Response
  • TurnOffRequest / Response
  • SetTargetTemperatureRequest / Confirmation

The DiscoverApplianceRequest directive is received at the lambda_handler function when I ask Alexa to discover devices (either via the Echo or through the app). The sample lambda_handler returns a DiscoverApplianceResponse with a single dummy device.

The most important fields in the device details are as follows:

"applianceId":"device001"
"friendlyName":"Smart Home Virtual Device"
"actions":["turnOn","turnOff"]
The applianceId must be unique and the friendlyName is the name that Alexa will recognise when you communicate via speech.

The actions indicate which directives the device supports. The Smart Home Skill will only route through directives listed in the actions. So, if a device doesn't indicate 'setTargetTemperature' in the actions field no amount of shouting at Alexa to set the temperature will route the message through to the lambda_handler.

Communicating between my Lambda function and my raspberry pi


I altered the sample code to return a friendlyName of 'water pump'. Asking Alexa to 'turn on the water pump' resulted in a TurnOnRequest routing through to the lambda_function and a TurnOnResponse being returned. Now it was time to add the actual action of instructing the raspberry pi / zway to turn on the pump.

I already had a load of python scripts that I had used for testing so just copied a few functions into my lambda handler and hoped for the best.

def set_pumpstatus (cookie,id,payload):

    headers={'accept':'application/json','Content-Type':'application/json','Cookie':'ZWAYSession='+cookie}
    # now set the pump status
    r = requests.post(pumpstatus_url+str(id),data=json.dumps(payload),headers=headers)

    return r

Unfortunately the Python requests library is not included in the AWS lambda environment so I was unable to use the 'edit code inline' feature. This turned out to be a minor inconvenience as I just needed to follow the instructions on Creating a Deployment Package (Python) for AWS and upload a zip file with my code in lambda_function.py and the necessary requests library.

# this handles Control directives from Alexa
# it determines which device to control from the id
def handleControl(context, event):
    payload = ''
    header = ''
    device_id = event['payload']['appliance']['applianceId']
    message_id = event['header']['messageId']
    control_type = event['header']['name']
    headerTemplate = {
                "namespace":"Alexa.ConnectedHome.Control",
                "payloadVersion":"2",
                "messageId": message_id
            }
 
    # is it a pump request
    if device_id == pumpDeviceID:
        if (control_type == 'TurnOnRequest') or (control_type == 'TurnOffRequest'):
            cookie = login.send()  
            if control_type == 'TurnOnRequest':
                pumpstatus.set_pumpstatus(cookie,1,{"data":"true"})
                headerTemplate['name'] = "TurnOnConfirmation"
             
            if event['header']['name'] == 'TurnOffRequest':
                pumpstatus.set_pumpstatus(cookie,1,{"data":"false"})
                headerTemplate['name'] = "TurnOffConfirmation"
            header = headerTemplate
            payload = { }
    return { 'header': header, 'payload': payload }  

Remarkably, this went smoothly and within minutes I could turn on and off the water pump with my voice.

Adding more devices and features


The 'devices' I require Alexa to manage are basically the water pump and the rooms (my heating zones). I left the water pump details hard coded into the lambda function but dynamically create the room list by requesting the information from my ZWay server.

The directives I am using for the rooms are as follows:


  • TurnOn / TurnOff - sets boost mode / returns to previous mode
  • SetTargetTemperature - sets boost mode and sets the boost temperature
  • SetPercentage - sets boost mode and uses the % value for the boost duration 1%=1 hour etc.


The lambda function can be found on github with my other zway python scripts.

And here is a demo of the Alexa app logging in to the Smart Home Skill, discovering devices and communicating with them. The status is refreshed on my Android app to show that the operation has been carried out.










Monday, February 27, 2017

Logging Data from my system

Most home automation systems seem to provide some data logging and graphing feature so I decided to look into adding this to my system.

Heat Genius Chart

British Gas Hive Chart


At this stage I am not sure how useful charts of data collected from my central heating system are going to be, but I was keen to learn some new skills. This is a record of my experimenting so far. I plan to add more posts as it develops.

What to log?


There is all sorts of sensor data coming out of my system but the things I am most interested in are:

  • actual temperatures of rooms v desired temperatures over time
  • how often the system is calling for heat
  • external temperature over time
  • actual boiler on/off status
The first three are easily available within my ZWay module. The 4th, the actual boiler status (ie whether it is running) is not available but I have an idea as to how I might measure it using an arduino - but that is a project for another day.

How to log?

The ZWay system already provides a module called SensorValueLogging and from the UI you can easily setup a log to a file or to an external service:

The choices of what to log are limited to actual sensors in the system so it isn't quite what I want. However, it was going to be a good starting point and the code that does it would be very simple to adapt for my own code. But how does it work?

Logging to a file


I activated the module to log the value of one of my thermostats to a JSONFile. The module binds to the value of the thermostat and logs every value change via this piece of code:

            var storedLog = loadObject("SensorValueLogging_" + vDev.id + "_" + self.id);
            if (!storedLog) {
                storedLog = {
                    deviceId: vDev.id,
                    deviceName: vDev.get("metrics:title"),
                    sensorData: []
                };
            }
            storedLog.sensorData.push({"time": Date.now(), "value": vDev.get("metrics:level")});
            saveObject("SensorValueLogging_" + vDev.id + "_" + self.id, storedLog);

This resulted in a SensorValueLogging json file appearing in the storage directory of my ZWay server:


inside I could see a JSON object containing {timestamp, value} pairs for my thermostat:


sensor data displayed via http://jsonviewer.stack.hu/


I could certainly mimic this operation in my own code but was a little puzzled by how loadObject() and saveObject() resulted in such strange file names. The source code was not easily available so I turned to the zwave forum and this post gave an explanation.


I had a quick look at what MD5 is and was satisfied that all I needed to know was that loadObject() and saveObject() would provide a mechanism for me to store and retrieve collected data locally on my rPi (if that is what I decided to do). Someone on the ZWave forum did this a couple of years ago and also provided some web pages to display the data (via Google Charts).

However, I was more keen to investigate sending the data into the cloud.

Logging to the cloud

A quick google search for cloud data logging services and I was overwhelmed by choice. Obviously most of these are commercial based for huge quantities of server data. Where to begin?

Then I came across this page from The Complex and Sustainable Urban Networks (CSUN) Laboratory at the University of Illinois at Chicago. They have been logging temperature and humidity data from an arduino since 2014. The data is logged to a Google Spreadsheet every 10 minutes (via PushingBox) and is then extracted and displayed with Google Charts. Perfect.

I signed up to PushingBox and reconfigured the SensorValueLogging module to send its readings to PushingBox:



The devid was just that of the demo scenario that is automatically created when you sign up. It was linked to a service that sent an email to me. I adjusted the email to pass on the temperature reading:


As with the JSON file example, SensorValueLogging binds to the value of the Thermostat and logs every change. But for HTTP logging it runs this piece of code:

        if (self.config.logTo === "HTTPGET") {
            http.request({
                method: 'GET',
                url: self.config.url.replace("${id}", vDev.id).replace("${value}", vDev.get('metrics:level'))
            });
        }

This did indeed end up sending me emails every time the temperature changed in my office:


PushingBox to Google Docs

Next step was to push the data to a Google Spreadsheet rather than to an email account.
I used this excellent instructables guide to help me through. The interface has changed a bit since that was written but the steps are all valid. Essentially I needed to make PushingBox mimic what happens when data is submitted via a Google Form to a Google Spreadsheet. So, I started by creating a form and submitting some data.
  • Create a google form here with a single question and a short text answer. 
  • Link the form to a spreadsheet (either existing or a new one) and make the spreadsheet shareable
  • Click on SEND and choose the 'send via link' option.
  • Copy the URL and paste it into a new browser window 
  • Submit a temperature reading via the form
The Instructables guide suggested examining the HTML to find the id of the input box. This does work, but I didn't really understand what I was doing with the information I retrieved. I ran Fiddler to trace the actual message the google forms sends when you press submit.

In my case, it sent the following when I submitted a value of 5.

The URL is the same as the one you see after pressing submit - the one that the Instructables guide asks you to copy.

POST https://docs.google.com/forms/d/e/1FAIpQLSdewH0LtfKOpCDqL_Ydi76dUYxrsZs41YDAHszyuvBstApP4g/formResponse HTTP/1.1

with

entry.463855442=5&fvv=1&draftResponse=%5B%2C%2C%227319728817704999679%22%5D%0D%0A&pageHistory=0&fbzx=7319728817704999679

in the body.

I set up a PushingBox service to attempt to do the same thing:



The full URL can be seen here.


I then added this service to the original demo scenario with the entry data from the body I had captured in Fiddler. $value$ is used to tell PushingBox to substitute the actual value given to it from the ZWay SensorValueLogging module. The rest of the data I had captured in the body I ignored.


And here are the resulting entries in my spreadsheet:


Which I then displayed via google charts thanks to a quick bit of hacking of the CSUN HTML and javascript.


Next step will be to write my own logging object / module and start collecting all of the data I require.

Friday, December 30, 2016

Adding a Fibaro Button to control my hot water pump

Controlling the hot water pump by activating a ZWave switch from my phone app is great, but it would be more convenient for everyone if this could also be operated by a simple button located close to where the water is needed.

The only ZWave button I could find was this one by Fibaro. 


It seemed overkill (and pricey) for my purposes, but I liked the look of it. It had not been on the market long and I wasn't sure that it was going to work with the ZWay software. But I took a chance and ordered it anyway.

Setting up the button


Including it into my ZWave network was straightforward. And it appeared in the SmartHome UI:


And here are some of the views from the Expert UI:

Interview Tab showing supported command classes

Configuration Tab
Association Tab


but I wasn't sure what to expect after that. According to the ZWave.me forum it is not yet supported by ZWay, but a couple of people had managed to catch a single click event which was all I needed. So I started fiddling with it and seeing what messages appeared.


What happens for a single button press?

Pressing the button once resulted in a notification appearing in the SmartHome UI:


Looking into the log file, the first message picked up after this button press was:

[2016-12-30 14:48:07.676] [D] [zway] RECEIVED: ( 01 08 00 04 00 28 02 98 40 01 )

followed by a series of SENDS and ACKS

and then what looked like the information I needed:

[2016-12-30 14:48:08.319] [D] [zway] SETDATA devices.40.instances.0.commandClasses.91.data.sequence = **********
[2016-12-30 14:48:08.319] [D] [zway] SETDATA devices.1.instances.0.commandClasses.91.data.srcNodeId = 40 (0x00000028)
[2016-12-30 14:48:08.320] [D] [zway] SETDATA devices.1.instances.0.commandClasses.91.data.srcInstanceId = 0 (0x00000000)
[2016-12-30 14:48:08.320] [D] [zway] SETDATA devices.1.instances.0.commandClasses.91.data.keyAttribute = 0 (0x00000000)
[2016-12-30 14:48:08.320] [D] [zway] SETDATA devices.1.instances.0.commandClasses.91.data.currentScene = 1 (0x00000001)
[2016-12-30 14:48:08.320] [D] [zway] SETDATA devices.40.instances.0.commandClasses.91.data.keyAttribute = 0 (0x00000000)
[2016-12-30 14:48:08.321] [D] [zway] SETDATA devices.40.instances.0.commandClasses.91.data.currentScene = 1 (0x00000001)
[2016-12-30 14:48:08.378] [I] [core] Notification: device-info (device-OnOff): {"dev":"Fibar Group (40.0.0.1) Button","l":"on"}

From the ZWave.me documentation command class 91 is a 'central scene command':



I did a little experimenting here and found that the following keyAttribute values resulted from Fibaro button presses:


press 1 time: 0x00
press 2 times: 0x03
press 3 times: 0x04
press 4 times: 0x05
press 5 times: 0x06
hold down: 0x02 followed by 0x01 once released

But really I was only interested in 'has it been pressed' rather than how many times or for how long. Still, it is good to know that even though the button is not officially supported it is possible to use all of its features. For my purposes I just needed to bind to changes in the keyAttribute.

Binding to the keyAttribute value

Here is my code:

        // find the hot water pump switch
         hotWaterSwitch = getDeviceByManufacturer(FIBARO_ID, FIBARO_BUTTON_ID, FIBARO_BUTTON_TYPE);
         console.log ("MYTESTMOD: hot water switch is " + hotWaterSwitch);
         if (hotWaterSwitch != -1) { 
            zway.devices[hotWaterSwitch].instances[0].commandClasses[91].data.keyAttribute.bind(
                                                      function() 
                                                      {
                                                        self.rooms[1].setPumpStatus(true);
                                                      });
         }
         else {
            controller.addNotification("error", "No Hot Water Switch Present", "module", "MyTestMod");
         }
         self.hotWaterSwitch = hotWaterSwitch;

And here is my button:



currently residing on top of my radio, close to the sink. Working great so far!

Friday, February 05, 2016

Choosing the z-wave TRVs

After initial investigations involving the Raspberry Pi, ZWave daughter card and Hostmann thermostat it was time to think about choosing the radiator valves. The two main competitors are the

Z-Wave Danfoss LC-13 Living Connect Radiator Thermostat


They appear to be equally respected across the home automation forums in terms of reliability, battery life and ease of connection. Aesthetically I prefer the Danfoss valves and it is these that HeatGenius adopted for their system. The only doubts I had were over problems that people reported with the early versions, but these seem to have been fixed with version 2. I decided I would purchase a single Danfoss valve to try out.

But how easy would it be to add the Danfoss valve to an existing system?

According to the manufacturer "The battery-powered thermostat is easily fitted to most radiator types in just a few minutes, " and online videos seemed to back this up. Further investigation revealed that it comes with two types of adapter

RA and M30 x 1.5(K) valve adapters

which is probably fine for the majority of UK homes. However, older houses like mine tend to have a mix of valves of various ages that have been added over time, so it was time to carry out a radiator audit.

The results of my radiator audit

17 radiators in total

3 old school radiators with no TRVs
2 radiators with Honeywell TRVs
6 radiators with Pro Heat TRVs
6 radiators with Giacomini TRVs



Honeywell TRV Giacomini TRV Pro Heat TRV

The Honeywell and Pro Heat have heads that screw on and that are interchangeable. I measured the width of the fitting and it was around 30mm. These are M30 heads.

The Giacomini TRV is very different. It is a push on type that is secured with 4 grub screws. I couldn't find anywhere selling such a valve but have since discovered similar valves branded as Barlo, Tower Boss & Harp. I think that they are quite old.

I was confident that the supplied M30 adapters would work with all but the Giacomini TRVs and ordered a single Danfoss LC-13 to get going with. It was easy to install this in place of both the Pro Heat and the Honeywell TRVs, but as suspected there was no suitable adapter for the Giacomini. Unfortunately the Living Room has two of these (and a Pro Heat) and as this was to be the first of my controllable rooms it was important to find a way of adapting or replacing these valves. I certainly did not want to be draining down the system to install new TRVs at such an early stage in the project.

A word on radiator adapters

Most TRVs sold in the UK today are of the M30 x 1.5mm variety. M30 refers to the 30mm measurement across the thread and 1.5mm is the pitch of the thread. Standardizing this fitting has allowed heads from other manufacturers to be fitted onto already installed bodies. However, many homes are fitted with older, less standard valves whose bodies will not work with M30 heads or whose pin size is incompatible.

This has led to the marketing of numerous valve adapters allowing M30 valve heads to replace older (perhaps malfunctioning) heads without the need to replace the whole body (with its associated installation costs).

It was my understanding that if a suitable adapter could be found then I could use this in addition to the M30 to Danfoss adapter supplied. So, the search was on:

Vesternet (from whom I purchased my Danfoss valve) sell a few adapters:

M28 - Comap
M28 - Herz
M28  - MMA or S&B

but none are compatible with my Giacominis.

The only place in the UK that I could find a suitable candidate was Heating Controls Online:

Click here to go to actual page
The picture is not at all clear so I contacted Heating Controls Online for advice. They were very quick to respond and assured me that this was what I was looking for.

I had also seen what looked like the same thing for sale on ebay and amazon (both from Germany). In Germany, the TRV brand is Heimeir but it looks identical to the Giacomini. And here is a picture of the adapter showing the part number as 9700-33.

HEIMEIER-Adapter-fur-Giacomini-Ventile-9700-33-700










With postage, they are no cheaper than those from the UK. At around £15 each, it is an expensive solution but I took a chance and ordered two from heatingcontrolsonline. On arrival I noted the same part number as those sold from Germany which gives an extra purchasing option in the future.

Fitting the Living Room TRVs

I removed the Giacomini heads from the living room and fitted the adapters. This was a bit fiddly as they are not screw on and require a tiny allen key. Once fitted I screwed the M30 adapter on top and fitted the Danfoss Living Connect. The fit is secure, but I am not sure the pin gets pushed completely to the bottom so it is possible they may not close completely. It is a good enough solution though.

By this stage I had had some time to experiment with controlling the single Danfoss LC via the ZWay software on the Raspberry Pi and was satisfied with the results. I ordered two more Danfoss TRVs and got ready to take control of the Living Room.

For anyone having issues finding suitable adapters, this handy chart lists out many TRVs around in the UK and HeatGenius have a useful online compatibility checker.