ESP32 Classic MQTT Publisher & Home Assistant

Started by Graham, February 13, 2020, 05:58:57 PM

Previous topic - Next topic

ClassicCrazy

Quote from: Graham on January 29, 2021, 08:52:15 AM
Quote from: ClassicCrazy on January 29, 2021, 12:29:47 AM
...
success on getting Android Classic Monitoring  App on MQTT 
( first had trouble getting mqtt working on my Fire android tablet - but after clearing , uninstalling, installing newest version , more stuff like restarting it I finally got it to work . Version says 3.0.6 . but I installed same APK on my galaxy 4 phone and it shows 3.1.0 version. )
Larry

Hmm, not sure why it's showing version 3.1.0, could your galaxy 4 already have had version 3.1.0, you would not be able to install 3.0.6 without first uninstalling 3.1.0.
The "MQTT" branch of ClassicMonitor is currently at version 3.0.6,
the new "master" branch (currently 3.1.4) is the merge of the latest updates of the app store branch "NoMQTT" with the MQTT branch as well as an experimental update to access my Tri-Star TS-45 using the modbus adapter.
Version 3.0.6 should be ok for testing ClassicMQTT


Graham

I probably had some other version of Classic Monitoring app on my phone already and didn't realize it- I don't use that phone too often as I have lineage OS on it and don't pay for phone service.

Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Graham

I updated the image on dockerHub https://hub.docker.com/r/classicdiy/classicmqtt so that you can now define the startup parameters in the environment section of the docker-compose.yml file when using IOTStack to setup your docker containers. You no longer need the .env file or the command: parameters.
Updated the wiki accordingly...
https://github.com/ClassicDIY/ClassicMQTT/wiki/3.1-Raspberry-Pi-setup-using-IOTStack


Graham
Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s

ClassicCrazy

#182
Quote from: Graham on January 31, 2021, 10:41:49 AM
I updated the image on dockerHub https://hub.docker.com/r/classicdiy/classicmqtt so that you can now define the startup parameters in the environment section of the docker-compose.yml file when using IOTStack to setup your docker containers. You no longer need the .env file or the command: parameters.
Updated the wiki accordingly...
https://github.com/ClassicDIY/ClassicMQTT/wiki/3.1-Raspberry-Pi-setup-using-IOTStack


Graham

I got this error on build - probably because I had previous build the other way ?
Issue running build:
while constructing a mapping
  in "./compose-override.yml", line 2, column 3
found duplicate key "classic_mqtt" with value "ordereddict([('container_name', 'classic_mqtt'), ('image', 'classicdiy/classicmqtt'), ('restart', 'unless-stopped'), ('environment', ['LOGLEVEL=DEBUG', 'CLASSIC=192.168.2.31', 'CLASSIC_PORT=502', 'CLASSIC_NAME=HoopPV', 'MQTT_HOST=mosquitto', 'MQTT_PORT=1883', 'MQTT_ROOT=ClassicMQTT', 'MQTT_USER=ClassicPublisher', 'MQTT_PASS=ClassicPub']), ('networks', ['iotstack_nw'])])" (original value: "ordereddict([('container_name', 'classic_mqtt'), ('image', 'classicdiy/classicmqtt'), ('restart', 'unless-stopped'), ('environment', ['LOGLEVEL=DEBUG', 'CLASSIC=192.168.3.2', 'CLASSIC_PORT=502', 'CLASSIC_NAME=CLASSIC2', 'MQTT_HOST=mosquitto', 'MQTT_PORT=1883', 'MQTT_ROOT=ClassicMQTT', 'MQTT_USER=ClassicPublisher', 'MQTT_PASS=ClassicPub']), ('networks', ['iotstack_nw'])])")
  in "./compose-override.yml", line 20, column 3

To suppress this check see:
    http://yaml.readthedocs.io/en/latest/api.html#duplicate-keys

Duplicate keys will become an error in future releases, and are errors
by default when using the new API.

I had already deleted .env file , then I deleted the classic part of docker_compose.yml , I used portainer and removed previous classicmqtt image, rebooted pi , but it still won't build and I get the same error as above.
After breakfast I may try to do what the link in the error message says to do if I can figure out how to do that.

Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Graham

Quote from: ClassicCrazy on January 31, 2021, 01:24:58 PM

I got this error on build - probably because I had previous build the other way ?
Issue running build:
while constructing a mapping
  in "./compose-override.yml", line 2, column 3
found duplicate key "classic_mqtt" with value "ordereddict([('container_name', 'classic_mqtt'), ('image', 'classicdiy/classicmqtt'), ('restart', 'unless-stopped'), ('environment', ['LOGLEVEL=DEBUG', 'CLASSIC=192.168.2.31', 'CLASSIC_PORT=502', 'CLASSIC_NAME=HoopPV', 'MQTT_HOST=mosquitto', 'MQTT_PORT=1883', 'MQTT_ROOT=ClassicMQTT', 'MQTT_USER=ClassicPublisher', 'MQTT_PASS=wt9m']), ('networks', ['iotstack_nw'])])" (original value: "ordereddict([('container_name', 'classic_mqtt'), ('image', 'classicdiy/classicmqtt'), ('restart', 'unless-stopped'), ('environment', ['LOGLEVEL=DEBUG', 'CLASSIC=192.168.3.2', 'CLASSIC_PORT=502', 'CLASSIC_NAME=CLASSIC2', 'MQTT_HOST=mosquitto', 'MQTT_PORT=1883', 'MQTT_ROOT=ClassicMQTT', 'MQTT_USER=ClassicPublisher', 'MQTT_PASS=ClassicPub']), ('networks', ['iotstack_nw'])])")
  in "./compose-override.yml", line 20, column 3
...
Larry

Looks like you have the container name in there twice

the second block should be called something different than the first and must have a unique container_name (see attached example)

you can delete your docker-compose.yml file in IOTStack, the menu.sh->Build Stack will create a new one with everything you selected.


Graham.
Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s

ClassicCrazy

#184
Quote from: Graham on January 31, 2021, 02:42:12 PM
Quote from: ClassicCrazy on January 31, 2021, 01:24:58 PM

I got this error on build - probably because I had previous build the other way ?
Issue running build:
while constructing a mapping
  in "./compose-override.yml", line 2, column 3
found duplicate key "classic_mqtt" with value "ordereddict([('container_name', 'classic_mqtt'), ('image', 'classicdiy/classicmqtt'), ('restart', 'unless-stopped'), ('environment', ['LOGLEVEL=DEBUG', 'CLASSIC=192.168.2.31', 'CLASSIC_PORT=502', 'CLASSIC_NAME=HoopPV', 'MQTT_HOST=mosquitto', 'MQTT_PORT=1883', 'MQTT_ROOT=ClassicMQTT', 'MQTT_USER=ClassicPublisher', 'MQTT_PASS=wt9m']), ('networks', ['iotstack_nw'])])" (original value: "ordereddict([('container_name', 'classic_mqtt'), ('image', 'classicdiy/classicmqtt'), ('restart', 'unless-stopped'), ('environment', ['LOGLEVEL=DEBUG', 'CLASSIC=192.168.3.2', 'CLASSIC_PORT=502', 'CLASSIC_NAME=CLASSIC2', 'MQTT_HOST=mosquitto', 'MQTT_PORT=1883', 'MQTT_ROOT=ClassicMQTT', 'MQTT_USER=ClassicPublisher', 'MQTT_PASS=ClassicPub']), ('networks', ['iotstack_nw'])])")
  in "./compose-override.yml", line 20, column 3
...
Larry

Looks like you have the container name in there twice

the second block should be called something different than the first and must have a unique container_name (see attached example)

you can delete your docker-compose.yml file in IOTStack, the menu.sh->Build Stack will create a new one with everything you selected.


Graham.

That seems to have done it Graham - thanks.
You may want to edit the file I got off the link .
This one - is where I got it from
https://github.com/ClassicDIY/ClassicMQTT/blob/master/code/Python/compose-override.yml

Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Graham

Quote from: ClassicCrazy on January 31, 2021, 03:00:55 PM
...

That seems to have done it Graham - thanks.
You may want to edit the file I got off the link .
This one - is where I got it from
https://github.com/ClassicDIY/ClassicMQTT/blob/master/code/Python/compose-override.yml

Larry

thanks Larry
I updated the file


Graham
Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s

ClassicCrazy

#186
Quote from: Graham on January 31, 2021, 03:08:19 PM
Quote from: ClassicCrazy on January 31, 2021, 03:00:55 PM
...

That seems to have done it Graham - thanks.
You may want to edit the file I got off the link .
This one - is where I got it from
https://github.com/ClassicDIY/ClassicMQTT/blob/master/code/Python/compose-override.yml

Larry

thanks Larry
I updated the file


Graham

Okay rebuilt worked .
I wasn't getting Classic2 publishing
Just found a typo I put in on the ip address
That took awhile for me to figure out !
Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

mcsarge

Hey All,

I am working on an update to the python implementation, it can read the environment variables directly in the code, so once the environment variable are set in the docker-compose or at the command line, it will be able to get those values in the python code. I am implementing it so it will use the environment variables if the value is not set in the command line. That way a command line setting will override the environment variable. I think the the whole creating of the .env file confused too many people. As soon as I get it finished and tested I will create a pull.

This may make the Home Assistance setup easier too.

Matt
Off Grid Island in Ontario Canada (Beaverstone Bay)
Primary: Classic 150 + wbjr; 3s2p HES 270watt
Winter: SolarBoost 50 MPPT (into wbjr); 2 x Sharp NE-80EJEA 80watt
Pack: 4s2p ROLLS S6-460AGM 6V for 24V pack
Inverter/Charger: Trace DR2424
Call Sign: KG4EUF

Graham

Quote from: mcsarge on February 16, 2021, 08:35:48 AM
Hey All,

I am working on an update to the python implementation, it can read the environment variables directly in the code, so once the environment variable are set in the docker-compose or at the command line, it will be able to get those values in the python code. I am implementing it so it will use the environment variables if the value is not set in the command line. That way a command line setting will override the environment variable. I think the the whole creating of the .env file confused too many people. As soon as I get it finished and tested I will create a pull.

This may make the Home Assistance setup easier too.

Matt

Hey Matt, did you see my pull request https://github.com/ClassicDIY/ClassicMQTT/pull/25 that adds the environment variables...


Graham.
Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s

mcsarge

Well,

No I did not, stupid me. I put a bunch of work into the validator set them first and your method of setting them in the initialization looks far easier! I will get your changes and then push them down to me.

Matt

Quote from: Graham on February 16, 2021, 08:52:10 AM
Quote from: mcsarge on February 16, 2021, 08:35:48 AM
Hey All,

I am working on an update to the python implementation, it can read the environment variables directly in the code, so once the environment variable are set in the docker-compose or at the command line, it will be able to get those values in the python code. I am implementing it so it will use the environment variables if the value is not set in the command line. That way a command line setting will override the environment variable. I think the the whole creating of the .env file confused too many people. As soon as I get it finished and tested I will create a pull.

This may make the Home Assistance setup easier too.

Matt

Hey Matt, did you see my pull request https://github.com/ClassicDIY/ClassicMQTT/pull/25 that adds the environment variables...


Graham.
Off Grid Island in Ontario Canada (Beaverstone Bay)
Primary: Classic 150 + wbjr; 3s2p HES 270watt
Winter: SolarBoost 50 MPPT (into wbjr); 2 x Sharp NE-80EJEA 80watt
Pack: 4s2p ROLLS S6-460AGM 6V for 24V pack
Inverter/Charger: Trace DR2424
Call Sign: KG4EUF

daklein

Thanks McSarge & Graham for this great tool.  I have some limited data available from my inverters, but with classicmqtt I'm also able to get battery bank voltage, which makes it possible to roughly control some loads and make use of variable solar production, instead of losing it as the main battery is near full.

I redid some of my RPi with ClassicMQTT & IOTstack, after the sd card filled up and things broke.   I believe I had previously built the docker image & was using the .env & command line argument method to give ClassicMQTT some of the parameters.  It worked. 

I followed these instructions (Thanks!),  https://github.com/ClassicDIY/ClassicMQTT/wiki/3.1-Raspberry-Pi-setup-using-IOTStack
This time I think I am running the image from DockerHub, portainer shows me that it was built 1/31   
classicdiy/classicmqtt:latest   120.3 MB   2021-01-31 08:56:57

It is working, but,  the former .env file & command line arguments for publish rate,  now in the environment section of compose-override.yml seem to not have an effect.     It publishes just at the default 5 minute rate, I'm not sure what I'm missing?  I'm pretty sure it's seeing the compose-override.yml file I'm editing,  if I change the classic's ip addr, etc, that will break it or not.   

If i change the override file, and run the ./menu.sh script to build and start the IOTstack,  it does recreate the classicmqtt container, and the main docker-compose.yml gets text changes.


Start Stack:
docker-compose up -d --remove-orphans
WARNING: Some networks were defined but are not used by any service: vpn_nw, iotstack_nw_internal, nextcloud_internal
nodered is up-to-date
portainer-ce is up-to-date
grafana is up-to-date
influxdb is up-to-date
Recreating classic_mqtt ...
mosquitto is up-to-date
Recreating classic_mqtt ... done

Stack Started




Here's what's in the top of docker-compose.yml,  from the compose-override.yml file that was edited.

pi@hassbian:~/IOTstack $ more docker-compose.yml
version: '3.6'
services:
  classic_mqtt:
    container_name: classic_mqtt
    image: classicdiy/classicmqtt
    restart: unless-stopped
    environment:
    - LOGLEVEL=DEBUG
    - CLASSIC=192.168.16.10
    - CLASSIC_PORT=502
    - CLASSIC_NAME=MNC250CP
    - MQTT_HOST=mosquitto
    - MQTT_PORT=1883
    - MQTT_ROOT=ClassicMQTT
    - MQTT_USER=mqtt_user
    - MQTT_PASS=mqtt_user_pw
    - AWAKE_PUBLISH_RATE=3
    - SNOOZE_PUBLISH_RATE=5
    networks:
    - iotstack_nw

# uncomment the following block if you have a second classic, change the CLASSIC ip address and CLASSIC_NAME
#  classic_mqtt2:
#    container_name: classic_mqtt2
#    image: classicdiy/classicmqtt
#    restart: unless-stopped
#    environment:
#      - LOGLEVEL=DEBUG
#      - CLASSIC=192.168.86.137
#      - CLASSIC_PORT=502
#      - CLASSIC_NAME=MyShed
#      - MQTT_HOST=mosquitto
#      - MQTT_PORT=1883
#      - MQTT_ROOT=ClassicMQTT
#      - MQTT_USER=ClassicPublisher
#      - MQTT_PASS=ClassicPub123
#    networks:
#      - iotstack_nw

# Here is an example using an external MQTT broker
#  classic_mqtt:
#    container_name: classic_mqtt
#    image: classicdiy/classicmqtt
#    restart: unless-stopped
#    environment:
#      - LOGLEVEL=DEBUG
#      - CLASSIC=192.168.86.37
#      - CLASSIC_PORT=502
#      - CLASSIC_NAME=MyWorkshop
#      - MQTT_HOST=192.168.86.82
#      - MQTT_PORT=1883
#      - MQTT_ROOT=ClassicMQTT
#      - MQTT_USER=HASSUser
#      - MQTT_PASS=HASSpw
#    networks:
#      - iotstack_nw
  grafana:
    container_name: grafana
    image: grafana/grafana
    restart: unless-stopped
    user: "0"
    ports:
    - "3000:3000"
    environment:
    - GF_PATHS_DATA=/var/lib/grafana
    - GF_PATHS_LOGS=/var/log/grafana
    volumes:
    - ./volumes/grafana/data:/var/lib/grafana
    - ./volumes/grafana/log:/var/log/grafana
    networks:
    - iotstack_nw

  python:
    container_name: python
    build: ./services/python/.
    restart: unless-stopped
    network_mode: host
    volumes:
    - ./volumes/python/app:/usr/src/app
  portainer-ce:

.  etcetera
.      etcetera
.           etcetera




Graham

Quote from: daklein on May 10, 2021, 05:57:09 PM
...
It is working, but,  the former .env file & command line arguments for publish rate,  now in the environment section of compose-override.yml seem to not have an effect.     It publishes just at the default 5 minute rate, I'm not sure what I'm missing?  I'm pretty sure it's seeing the compose-override.yml file I'm editing,  if I change the classic's ip addr, etc, that will break it or not.   


I pushed a new image to dockerHub, try re running "Build Stack" and "Start Stack" in IOTStack/menu.sh
I also had to set the SNOOZE_PUBLISH_RATE to 300

check the logs in portainer for classic_mqtt to verify the startup arguments

let me know if this works...
Graham.


Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s

daklein

Quote from: Graham on May 11, 2021, 10:56:59 AM

I pushed a new image to dockerHub, try re running "Build Stack" and "Start Stack" in IOTStack/menu.sh
I also had to set the SNOOZE_PUBLISH_RATE to 300

check the logs in portainer for classic_mqtt to verify the startup arguments

let me know if this works...
Graham.

Graham,   Thanks!    It does work.  It got the new image from dockerhub, and it's publishing at the faster rate I was looking for.

I set both the wake & snooze interval to 3 sec, and it seems to be populating into influxdb at that rate now.

The log shows it going in & out of snooze mode, opening and closing the connection to the Classic.  Is that ok, or likely to cause some issue?   Is snooze meant to be nighttime, no solar production, for time > WAKE_DURATION.
   
(An issue beyond the need to downsample longer term data in database, which I need to figure out eventually.  But I want the relatively faster rate to use for control, using battery voltage vs. target voltage.)

2021-05-11 21:31:07,934:DEBUG:classic_mqtt:Closing the modbus Connection, we are in Snooze mode
2021-05-11 21:31:07,935:DEBUG:classic_mqtt:Got data from Classic at 192.168.16.10:502
2021-05-11 21:31:07,938:DEBUG:classic_mqtt:Publishing: ClassicMQTT/MNC250CP/stat/readings
2021-05-11 21:31:07,940:DEBUG:classic_mqtt:Publishing: ClassicMQTT/MNC250CP/stat/info
2021-05-11 21:31:13,920:DEBUG:classic_mqtt:Opening the modbus Connection
2021-05-11 21:31:13,940:DEBUG:classic_mqtt:Closing the modbus Connection, we are in Snooze mode
2021-05-11 21:31:13,941:DEBUG:classic_mqtt:Got data from Classic at 192.168.16.10:502
2021-05-11 21:31:13,945:DEBUG:classic_mqtt:Publishing: ClassicMQTT/MNC250CP/stat/readings
2021-05-11 21:31:13,949:DEBUG:classic_mqtt:Publishing: ClassicMQTT/MNC250CP/stat/info
2021-05-11 21:31:19,923:DEBUG:classic_mqtt:Opening the modbus Connection
2021-05-11 21:31:19,942:DEBUG:classic_mqtt:Closing the modbus Connection, we are in Snooze mode
2021-05-11 21:31:19,944:DEBUG:classic_mqtt:Got data from Classic at 192.168.16.10:502
2021-05-11 21:31:19,948:DEBUG:classic_mqtt:Publishing: ClassicMQTT/MNC250CP/stat/readings




Graham

Quote from: daklein on May 11, 2021, 05:48:50 PM
...
The log shows it going in & out of snooze mode, opening and closing the connection to the Classic.  Is that ok, or likely to cause some issue?   Is snooze meant to be nighttime, no solar production, for time > WAKE_DURATION.
   

SNOOZE_PUBLISH_RATE should be around 300 seconds since, in this mode, it will disconnect from modbus after every read to allow the local app to get in and access the classic.
If you want it to stay awake you can send a MQTT cmnd with a json payload containing "stayAwake"

ClassicMQTT/MNC250CP/cmnd {"stayAwake"}
Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s