ESP32 Classic MQTT Publisher & Home Assistant

Started by Graham, February 13, 2020, 05:58:57 PM

Previous topic - Next topic

mcsarge

Graham,

OK, I have the nearly final, ready to be public version all checked in and ready.

Thanks,

Matt
Off Grid Island in Ontario Canada (Beaverstone Bay)
Primary: Classic 150 + wbjr; 3s2p HES 270watt
Winter: SolarBoost 50 MPPT (into wbjr); 2 x Sharp NE-80EJEA 80watt
Pack: 4s2p ROLLS S6-460AGM 6V for 24V pack
Inverter/Charger: Trace DR2424
Call Sign: KG4EUF

Graham

Quote from: mcsarge on March 02, 2020, 02:07:15 PM
Graham,

OK, I have the nearly final, ready to be public version all checked in and ready.

Thanks,

Matt

Ok, I merged it

ordered another RPI so I can try it out and start to learn python...


Graham.

Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s

mcsarge

You can actually run Python on a Mac or a Windoze PC, you just have to install it. I like to us VSCode to develop in, and from there you can launch the program easily if all is installed. While developing. I used ngrok to get to port 502 of my Classic up on the island, but now that it is running on my pi up there, I use ngrok to get to MQTT installed on the Pi - and I use 0.tcp.ngrok.io and that port in the Android App to look at the data.

Now that I have the infrastructure installed, I am going to work on a large LCD or LED display using an ESP32 for the wall of the cabin that will show some of the data from the Classic like battery voltage, WhizBangJr current and SOC, etc. It will have a button that will send a "wake" to get data more often, but for most applications, updates every 5 minutes (snoozing) will be just fine.

Matt
Off Grid Island in Ontario Canada (Beaverstone Bay)
Primary: Classic 150 + wbjr; 3s2p HES 270watt
Winter: SolarBoost 50 MPPT (into wbjr); 2 x Sharp NE-80EJEA 80watt
Pack: 4s2p ROLLS S6-460AGM 6V for 24V pack
Inverter/Charger: Trace DR2424
Call Sign: KG4EUF

ClassicCrazy

It will be interesting to see your display when you get it going Matt.
That is what I have been wanting to do here - just be able to see certain selected data from the Classic on a small LED display on the wall.
I know I could hang the MNGP up - but that is too much data and have to do bunch of button pushes.
I guess if I had to pick a few vital clues it would be
Charge state ,SOC,  system amps, target voltage ( temperature compensated voltage) and the battery voltage. Oh one more might be the Absorb timer . Those are the main things I usually look for at a glance .
Oh and if you want to make a big LED display here is an idea https://youtu.be/fz2QAV9z_o8

Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Graham

Quote from: mcsarge on March 02, 2020, 03:04:39 PM
You can actually run Python on a Mac or a Windoze PC, you just have to install it. I like to us VSCode to develop in, and from there you can launch the program easily if all is installed. While developing. I used ngrok to get to port 502 of my Classic up on the island, but now that it is running on my pi up there, I use ngrok to get to MQTT installed on the Pi - and I use 0.tcp.ngrok.io and that port in the Android App to look at the data.


Is your MQTT broker local on the Pi or in Azure (or any other cloud based broker)?

+ we could "Containerize" the python app using docker that would allow it to run anywhere
https://www.wintellect.com/containerize-python-app-5-minutes/


Graham.
Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s

mcsarge

Pretty funny you say that  as I am working on a Docker container that has Python3 on it and can run the App and will also have Mosquito running on it.

I have used both the Azure MQTT (I still have it up and running) but currently I have it using a Mosquito running on the Pi.

Matt
Off Grid Island in Ontario Canada (Beaverstone Bay)
Primary: Classic 150 + wbjr; 3s2p HES 270watt
Winter: SolarBoost 50 MPPT (into wbjr); 2 x Sharp NE-80EJEA 80watt
Pack: 4s2p ROLLS S6-460AGM 6V for 24V pack
Inverter/Charger: Trace DR2424
Call Sign: KG4EUF

mcsarge

Graham,

I have checked in the docker and docker-compose for the tool and created a pull request. Pretty slick, if I do say so myself.

So all you need is a clean Raspberry Pi, install docker and docker-compose, get the Python folder onto the Pi, create your .env file and run docker-compose and you are in business. I documented all this in the README.

Matt
Off Grid Island in Ontario Canada (Beaverstone Bay)
Primary: Classic 150 + wbjr; 3s2p HES 270watt
Winter: SolarBoost 50 MPPT (into wbjr); 2 x Sharp NE-80EJEA 80watt
Pack: 4s2p ROLLS S6-460AGM 6V for 24V pack
Inverter/Charger: Trace DR2424
Call Sign: KG4EUF

ClassicCrazy

Quote from: mcsarge on March 05, 2020, 12:20:30 PM
Graham,

I have checked in the docker and docker-compose for the tool and created a pull request. Pretty slick, if I do say so myself.

So all you need is a clean Raspberry Pi, install docker and docker-compose, get the Python folder onto the Pi, create your .env file and run docker-compose and you are in business. I documented all this in the README.

Matt
Let me know if you want me to test it out sometime.
I have raspberry pi 3 and can reflash it .

Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Graham

Quote from: mcsarge on March 05, 2020, 12:20:30 PM
Graham,

I have checked in the docker and docker-compose for the tool and created a pull request. Pretty slick, if I do say so myself.

So all you need is a clean Raspberry Pi, install docker and docker-compose, get the Python folder onto the Pi, create your .env file and run docker-compose and you are in business. I documented all this in the README.

Matt

Thanks Matt, I merged the code on github.
I'll give it a try when I get my new raspberry Pi 4


Graham.
Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s

mcsarge

Quote from: ClassicCrazy on March 05, 2020, 10:47:14 PM
Quote from: mcsarge on March 05, 2020, 12:20:30 PM
Graham,

I have checked in the docker and docker-compose for the tool and created a pull request. Pretty slick, if I do say so myself.

So all you need is a clean Raspberry Pi, install docker and docker-compose, get the Python folder onto the Pi, create your .env file and run docker-compose and you are in business. I documented all this in the README.

Matt
Let me know if you want me to test it out sometime.
I have raspberry pi 3 and can reflash it .

Larry

Larry,

Have at it, it is particular to Stretch, but a small change can support other distros of Raspbarian - There is an issue with running Docker on the latest version of Rasparian, so I am sticking to stretch.

Matt
Off Grid Island in Ontario Canada (Beaverstone Bay)
Primary: Classic 150 + wbjr; 3s2p HES 270watt
Winter: SolarBoost 50 MPPT (into wbjr); 2 x Sharp NE-80EJEA 80watt
Pack: 4s2p ROLLS S6-460AGM 6V for 24V pack
Inverter/Charger: Trace DR2424
Call Sign: KG4EUF

ClassicCrazy

Here is what I have done so far
I am using Pi3B+  model
Installed latest Raspian Buster ( stretch version not on pi download page so giving the Buster version a try)
set up pi for using ssh, vnc, using the whole sd card , and locale
updated and upgraded pi
Installed Docker and Docker compose according to
https://dev.to/rohansawant/installing-docker-and-docker-compose-on-the-raspberry-pi-in-5-simple-steps-3mgl
Noticed that some people in comments on above link said they had problems because default for docker compose was python 2
I followed directions on below link to change default to docker 3
https://learn.sparkfun.com/tutorials/python-programming-tutorial-getting-started-with-the-raspberry-pi/configure-your-pi
I reinstalled docker compose just to make sure it was correct
Now on to Matt's directions in Read Me file
install pymodbus on pi ( think I installed it for both python 2 and python 3 ?) using info on next link
https://www.raspberrypi.org/forums/viewtopic.php?t=251917
install paho mqtt using this command    pip install paho-mqtt
this was the paho mqtt info i used
http://www.steves-internet-guide.com/into-mqtt-python-client/

From Matts instructions
Install docker and docker-compose on your Rasberry Pi - look this up on the web and follow the instructions.( done as above)

Copy the repostory (if you understand git, you can get it that way too)
wget https://github.com/graham22/ClassicMQTT/archive/master.zip ( ran this )
Extract the zip file:
unzip master.zip ( ran this )
Change directory
cd ./ClassicMQTT/code/Python ( this step gets me lost - I get this error )
-bash: cd: ./ClassicMQTT/code/Python: No such file or directory
I think this step should say  cd ClassicMQTT-master/code/Python   ( is this correct ?)
Then I did
nano .env  ( to create and pasted in contents below with my classic ip address , etc ) 

Create the .env file to look like this:
CLASSIC_HOST=<IP address or URL>
CLASSIC_PORT=<Port usually 502>
CLASSIC_ROOT=<The MQTT Root, usually ClassicMQTT>
USER_ID=ClassicPublisher
USER_PASS=ClassicPub123


User docker-compose to start up the both the mosquitto and the script.
docker-compose -f classic_mqtt_compose.yml up

did the command above it is seems to work - no errors anyway - I saw it loading up : Downloaded newer image for python:3.7.6-slim-stretch
Should I change something in the code to download buster since I am using that and not stretch ?
I am not exactly sure the next step of what I put in my android to subscribe to this
I tried subscribing to 172.18.0.3 on port 1883 but I don't see it . My pi is plugged into the router - no wifi connected yet - and my android is wifi to my router . 
Screen shot below of my pi data
Calling it a night now but I think I had some success for not knowing much about what I am doing on Linux stuff.
Little things like that cd command would throw me off for good but tonight I think I figured out what to do .

Larry


system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Graham

#41
Quote from: mcsarge on March 05, 2020, 12:20:30 PM
Graham,

I have checked in the docker and docker-compose for the tool and created a pull request. Pretty slick, if I do say so myself.

So all you need is a clean Raspberry Pi, install docker and docker-compose, get the Python folder onto the Pi, create your .env file and run docker-compose and you are in business. I documented all this in the README.

Matt

Still waiting for my argon case for my raspberry pi 4 (https://www.argon40.com/catalog/product/view/id/52/s/argon-one-raspberry-pi-4-case/category/4/)
so I decided to try this using docker desktop on Windows 10 pro using the following yaml file that uses my existing mosquito MQTT  broker.

command:

docker-compose -f classic_mqtt_compose_nomosquito.yml up

classic_mqtt_compose_nomosquito.yml content:

version: "3"
services:
  classic_mqtt:
    image: classic_mqtt
    build: .
    environment:
      - LOGLEVEL=DEBUG
    networks:
      - localnet
    command: "--classic 192.168.86.37 --classic_port 502 --mqtt 192.168.86.82 --mqtt_root ClassicMQTT --user HASS --pass XXXX"
networks:
  localnet:

I was then able to view the data using the MQTT version of my android app.
Docker desktop for Windows 10 PRO (doesn't install on Windows 10 Home edition) : https://hub.docker.com/editions/community/docker-ce-desktop-windows

Also tried it on Ubuntu 19.04
Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s

mcsarge

#42
Congratulations (Larry)!

You are up and running (I can see it is getting info from your Classic) To use the flow, go into Graham's app and configure it to be an MQTT Subscriber and set the IP address of the MQTT server your Pi's IP address and port 1883. It should connect and off you go.

If your system is behind a firewall, you can either port forward that port in your router or install ngrok and forward the TCP port with the following command:
./ngrok 1883

So the beauty of using docker is that you do not need to install pymodbus or paho-mqtt. or even python at all! It all gets installed in the container for you and saved (I have directions for 2 modes, so I will fix that to be more clear). I will also fix the cd instruction (I forgot that the "-master" gets in there. You can change your python in the "Dockerfile" to be FROM python:3.7.6-slim-buster if you want but since they rely on the same architecture, it does not make much difference.

Thanks for testing!

I am now throwing together a website written in python using flask and socketio to connect to the MQTT server and display some of the readings like Battery Voltage, SOC, Current, etc. Once I get the simple monitor done, I will add in code to save the readings every 5 minutes in a database so I can show charts that cover readings over a period of time in chart fashion.

Matt
Off Grid Island in Ontario Canada (Beaverstone Bay)
Primary: Classic 150 + wbjr; 3s2p HES 270watt
Winter: SolarBoost 50 MPPT (into wbjr); 2 x Sharp NE-80EJEA 80watt
Pack: 4s2p ROLLS S6-460AGM 6V for 24V pack
Inverter/Charger: Trace DR2424
Call Sign: KG4EUF

mcsarge

Quote from: Graham on March 07, 2020, 08:21:39 AM
Quote from: mcsarge on March 05, 2020, 12:20:30 PM
Graham,

I have checked in the docker and docker-compose for the tool and created a pull request. Pretty slick, if I do say so myself.

So all you need is a clean Raspberry Pi, install docker and docker-compose, get the Python folder onto the Pi, create your .env file and run docker-compose and you are in business. I documented all this in the README.

Matt

Awesome!

I will add the no_mosquitto yaml file in for people who have mqtt already installed and running.

Matt

Still waiting for my argon case for my raspberry pi 4 (https://www.argon40.com/catalog/product/view/id/52/s/argon-one-raspberry-pi-4-case/category/4/)
so I decided to try this using docker desktop on Windows 10 pro using the following yaml file that uses my existing mosquito MQTT  broker.

command:

docker-compose -f classic_mqtt_compose_nomosquito.yml up

classic_mqtt_compose_nomosquito.yml content:

version: "3"
services:
  classic_mqtt:
    image: classic_mqtt
    build: .
    environment:
      - LOGLEVEL=DEBUG
    networks:
      - localnet
    command: "--classic 192.168.86.37 --classic_port 502 --mqtt 192.168.86.82 --mqtt_root ClassicMQTT --user HASS --pass XXXX"
networks:
  localnet:

I was then able to view the data using the MQTT version of my android app.
Docker desktop for Windows 10 PRO (doesn't install on Windows 10 Home edition) : https://hub.docker.com/editions/community/docker-ce-desktop-windows
Off Grid Island in Ontario Canada (Beaverstone Bay)
Primary: Classic 150 + wbjr; 3s2p HES 270watt
Winter: SolarBoost 50 MPPT (into wbjr); 2 x Sharp NE-80EJEA 80watt
Pack: 4s2p ROLLS S6-460AGM 6V for 24V pack
Inverter/Charger: Trace DR2424
Call Sign: KG4EUF

ClassicCrazy

#44
Matt ,
I don't have a firewall - but my android is still not connecting
I set the Grahams Classic Monitoring to
Mqtt subscriber
Mqtt broker host name to the router ip address that the pi has which is 192.168.3.29
mqtt broker port to 1883

I used the same mqtt user and passwords and root topic as you had in examples.

But I still don't connect with android.

Do I have to edit the mqtt host file  somewhere ?- - it says 127.0.0.1 is default in the code file
Also wondering if I should start out fresh again in case those extra copies of pymod and mosquitto I installed outside of docker are messing anything up ?

Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable