Migration off the Xiaomi Gateway using Zigbee2MQTT

When I first started with Home Assistant I kept reading about a Smart Home Ecosystem made by Xiaomi. I was amazed at how cheap they were compared to others in the market and was pretty impressed with how simple and clean they looked. The fact they were made in China with a cloud based platform was a concern and so was the price point they were just too cheap to be of any decent build quality or sensor accuracy. My curiosity got the better of me and I decided to purchase their home starter kit it was around $100AUD so no great loss if they turned out to be cheap crap, when they arrived I was immediately impressed with the packaging. When I cracked them open I could not believe just how well there simple design looked and felt and once integrated into Home Assistant how reasonably accurate they were. Half of my concerns had been addressed but there was still the little matter of a Chinese based cloud platform to deal with, my smart home philosophy is simple it has to be open source, flexible and work without the internet.

My first step was to block any external network access to my Xiaomi Gateway, this allowed the Ecosystem to continue to function without the concern of my data leaving my controlled space. Next I was happy to start flooding my house with as many Xiaomi sensors I could get my hands on, and stumbled across my first limitation of the Gateway its 30 connected devices limit. I thought no problem I wanted to relocated the gateway to use its built led as a TV backlight but if located there it didn’t have the coverage for my whole house. I jumped online and purchased a second gateway along with a bunch more sensors (yes I have a well documented Xiaomi addiction). I managed to setup the second device without much troubles and located it behind my PC monitor so now I had well positioned Gateways that also served as TV and Monitor backlight LED’s.

I now had a solution where I was able to use the Xiaomi Ecosystem without them needing to be connected to the outside world and through Home Assistant I was able to add and remove devices without having to use their horrible App, all was right in the world. I grew my Smart Home with Xiaomi as its spine and decided to share my work along the way, every single time I would have a comment “great setup man, but why don’t you use ConBee, deCONZ or Zigbee2MQTT?”. I looked into it did my research but couldn’t justify moving off my current setup as all of their benefits I had catered for No Cloud Check, No APP Check and Increased Range (I had sufficient with 2 Gateways) I just could not see the benefit of purchasing another Zigbee Coordinator having to flash it with custom firmware (Not something that concerns me that much I have been flashing hacked firmware since 2004) and potentially mess with a perfectly working network.

I continued to be asked that old question “why don’t you remove your proprietary gateway?” Well CTRL + C above man, I don’t have the time I would rather focus on other more pressing Smart Home additions like the boxes full of stuff I have sitting in my garage I just had to have but have not yet found the time to install/configure/integrate yet. I continued on with my Smart Home Life not giving Zigbee2MQTT another thought until I came across a post in the Home Assistant Forum Buy a ready2use zigbee2mqtt stick – flashed, antenna mod and printed case. OK I’ll bite, post creator Hans (h4nc) has phrased it as a series of questions and much to my surprise I found myself answering yes to all of them. His post also shows pictures of the units and their impressive 3D printed cases, but the thing that sold it for me was his attention to RF design principles and the way he attached the PCB mounted SMA connector. This design allows for connection of an external higher gain antenna extending the somewhat limited range of the original on board antenna design. I still had some questions about the various models and configurations he was able to supply, he answered me promptly with very specific answers and before I knew it I had ordered both a flashed coordinator and router. The devices were flashed, soldered, case printed, packaged and shipped the next day from Austria and I received them 19 days later in Australia well protected and with really high quality 3D printed cases.

 

Now that I have the core of my new Zigbee Network let’s have a closer look at what the zigbee2mqtt project is and what benefits one might gain by using it over the proprietary Zigbee Gateways supplied by vendors like Xiaomi, Philips Hue, Smartthings and IKEA. Essentially zigbee2mqtt allows you to get rid of your proprietary Zigbee Bridges, by doing this you can firstly save money by just having to purchase the bulbs, sensors & switches from the vendors you want without purchasing an expensive Bridge. This allows you to make up your own Zigbee Ecosystem comprised of devices from multiple vendors and the beauty of it is you can also do away with the 5 different apps you would have needed to use them all. Zigbee2MQTT does exactly that it bridges Zigbee devices messages to MQTT, which allows you to add new devices and control them very easily but also makes integrating with Home Assistant a piece of cake.

Lets now have a closer look at the makeup of a Zigbee network to better understand what hardware is required in order to run zigbee2mqtt. Zigbee has 3 different types of devices: Coordinator, Router and the End Device.  The key difference between these is that an end device can not route traffic, routers can route traffic, and the coordinator, in addition to routing traffic, is responsible for forming the network in the first place. Every network must have one and only one coordinator. To do that, it must select the appropriate channel, PAN ID, and extended network address.

Right so I have my Coordinator a CC2531 which has a device limit of 15 and another CC2531 as a Router which has a device limit of 21 as the router uses up one of the child devices of the Coordinator which leaves me with 14 + 21 = 35 connected end devices. Well doing the maths on my sensors around the house I have well over 50 however most AC powered devices like light bulbs serve as Routers also which will boost the number of supported devices well beyond my requirement. As stated above I purchased my quality items already flashed, you can however flash the devices yourself following the guides available here. Zigbee2MQTT maintainer Koen Kanters has provided multiple methods for running Zigbee2MQTT with great documentation for use with Linux, Docker and in a Virtual Environment. For those that run HASSIO, Daniel Welch has created a Zigbee2MQTT Addon and for the purposes of this article this is the method I will take you through.

Head over to the Hass.io panel section and click on the addon store tab, if you haven’t already add Daniel Welch’s addon repository once added scroll down to the newly added repo and you will be presented with 2 options zigbee2mqtt and zigbee2mqtt-edge. The zigbee2mqtt addon runs the most recent stable release of the platform and most recommended, the zigbee2mqtt-edge addon will run the latest development release I would only use this one if there is an update you simply must have before it is added to the main release.

Once the addon is installed we will need to configure some of the global variables within the addon config

 

Required Configuration Parameters

data_path – Set this to the path you’d like the add-on to persist data. Must be within the /share directory. Defaults to /share/zigbee2mqtt.

homeassistant – Set this to true if you want MQTT autodiscovery.

To achieve the best possible integration (including MQTT discovery):

  • In your Zigbee2mqtt configuration.yaml set homeassistant: true
  • In your Home Assistant configuration.yaml:
    mqtt:
    discovery: true
    broker: [YOUR MQTT BROKER]  # Remove if you want to use builtin-in MQTT broker
    birth_message:
      topic: 'hass/status'
      payload: 'online'
    will_message:
      topic: 'hass/status'
      payload: 'offline'
    

Mind you that if you want to use the embedded broker of Home Assistant you have to follow this guide.

Zigbee2mqtt is expecting Home Assistant to send it’s birth/will messages to hass/status. Be sure to add this to your configuration.yaml if you want Zigbee2mqtt to resend the cached values when Home Assistant restarts.

permit_join – Recommended to leave this to false and use runtime pairing. Set this to true when you setup new devices – make sure you set it back to false when done.

mqtt_server – The MQTT server address. Make sure you include the protocol. Example: mqtt://homeassistant

  • Depending on your configuration, the MQTT server URL will need to include the port, typically 1883 or 8883 for SSL communications. For example, mqtt://homeassistant:1883.

mqtt_base_topic – Prefix for your MQTT topic

serial_port – Serial port for your CC2531 stick.

  • To find out which serial ports you have exposed go to Hass.io > System > Host system > Show Hardware

network_key – Specify an encryption key for your zigbee network. See this section of the Zigbee2mqtt docs for more details and example usage. Leave this option empty (i.e. leave the default) if you don’t want to use a network key. Note that changing this requires repairing of all device connections. Hex values are not supported, only integer values (although hex values can be used by manually editing your configuration.yaml file and disabling overwrite).

Device-specific Configuration

Zigbee2mqtt allows certain device-specific configuration options. These can be configured in the add-on via the devicesoption. If you aren’t using this feature, you can leave the devices option empty as it appears by default. See below for an example device-specific configuration entry, adapted from the Zigbee2mqtt docs:

"devices": [
  {
    "id": "0x00158d0001d82999",
    "friendly_name": "my_occupancy_sensor",
    "retain": true,
    "occupancy_timeout": 20,
    "qos": 1
  }
]

temperature_precisionhumidity_precision, and pressure_precision are also supported.

Pairing

The suggested way to pair your devices is to enable zigbee2mqtt’s permit_join option from within Home Assistant using MQTT rather than through the add-on’s User Interface. Below is an example configuration that will allow you to enable and disable device pairing from the Home Assistant front end:

screen shot 2018-06-02 at 14 41 42

mqtt:
  broker: homeassistant # This will have to be your mqtt broker
  discovery: true

input_boolean:
  zigbee_permit_join:
    name: Allow devices to join
    initial: off
    icon: mdi:cellphone-wireless

timer:
  zigbee_permit_join:
    name: Time remaining
    duration: 600 # Updated this to the number of seconds you wish

sensor:
  - platform: mqtt
    name: Bridge state
    state_topic: "zigbee2mqtt/bridge/state"
    icon: mdi:router-wireless

automation:
  - id: enable_zigbee_join
    alias: Enable Zigbee joining
    hide_entity: true
    trigger:
      platform: state
      entity_id: input_boolean.zigbee_permit_join
      to: 'on'
    action:
    - service: mqtt.publish
      data:
        topic: zigbee2mqtt/bridge/config/permit_join
        payload: 'true'
    - service: timer.start
      data:
        entity_id: timer.zigbee_permit_join
  - id: disable_zigbee_join
    alias: Disable Zigbee joining
    trigger:
    - entity_id: input_boolean.zigbee_permit_join
      platform: state
      to: 'off'
    action:
    - data:
        payload: 'false'
        topic: zigbee2mqtt/bridge/config/permit_join
      service: mqtt.publish
    - data:
        entity_id: timer.zigbee_permit_join
      service: timer.cancel
    hide_entity: true
  - id: disable_zigbee_join_timer
    alias: Disable Zigbee joining by timer
    hide_entity: true
    trigger:
    - platform: event
      event_type: timer.finished
      event_data:
        entity_id: timer.zigbee_permit_join
    action:
    - service: mqtt.publish
      data:
        topic: zigbee2mqtt/bridge/config/permit_join
        payload: 'false'
    - service: input_boolean.turn_off
      data:
        entity_id: input_boolean.zigbee_permit_join

And add something like the following to your lovelace YAML file (if using YAML mode):

type: entities
entities:
  - entity: input_boolean.zigbee_permit_join
  - entity: timer.zigbee_permit_join
  - entity: sensor.bridge_state
show_header_toggle: false
title: Zigbee

Notes:

  • permit_join will be enabled for 10 minutes (based on code automation)

I have been running this setup for a few weeks now and have experienced significant Zigbee network improvements with stability, frequency interference mitigation and most of all coverage and reliability most in part to the wonderful job h4nc has done with the addition of higher gain external omni directional antennas installed on his customized CC2531.

The pairing process for Xiaomi products I found to be a little clunky at first but seeing as I needed to pair well over 50 devices to bring them over I managed to get plenty of practice and once you have the hang of it the process is less cumbersome then if you were to add them with the hybrid english/chinese native Xiaomi Home App. The limitations of the connected devices to the CC2531 was mitigated by adding a second in router mode and this was further extended by other devices being able to act as routers.

I have added several new Zigbee sensors to my network without ever having to touch a third party app or use a third party gateway which at the end of the day is really what having a “Smart Home” should really be about.

Comments

  1. tim Reply

    great article Jim, i was going to go the conBee deconz method to transfer all my xiaomi devices over but might go this way instead. its a little daunting for me but hopefully this guide will help

    • James McCarthy Reply

      Hi Tim,

      It was a pretty easy process in the end, it did take some time due to the amount of Zigbee sensors I had out there so if you can do it before deploying too many it will make life easier.

Leave a Reply

Your email address will not be published.