Installing Node Red on Raspberry PI Bullseye also Telegraf and Grafana

Started by ClassicCrazy, November 28, 2021, 04:13:01 PM

Previous topic - Next topic

ClassicCrazy

I want to try Jose's node red flow from this post.
http://midniteftp.com/forum/index.php?topic=4854.msg52373#msg52373
So I thought I would start out with fresh Raspberry Pi OS install. The newest one is called Bullseye replacing Buster.
First thing I ran into was I was running headless - pi connected via wireless and then using my windows computer VNC Viewer to get into the Pi's desktop.
The desktop was not showing up full monitor.
I found this youtube video which showed how to fix this issue ( took me awhile to find this solution)
https://youtu.be/-7rlLJmNmfA
Node Red came already installed in the Bullseye version I had used . But when I ran it there were some version errors shown.
I found this after some searching https://nodered.org/docs/getting-started/raspberrypi
From the link I ran this in a terminal window
bash <(curl -sL https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered)
That seems to have fixed the version issues that I was seeing before.
I imported Jose's flow
I  had to install node-red-contrib-modbus to pallete to get the modbus features for node-red
Also I had to install node-red-contrib-influxdb because Jose's sketch also uses that.
I need to install influxdb and grafana so decided to try to do it the easy way with IotStack project that uses Docker . I will find out if that is easy or not !
from this page https://sensorsiot.github.io/IOTstack/Getting-Started/
Follow New Installation on above link
Install all went fine for me. Restart
CD into IotStack  then run    ./menu.sh
Prompted to do some updates - did those all is well
Found out some of the updates for IOTstack not needed since they were already up to date but got prompted anyway.
This one sort of messed up my pi
"â", Docker version is currently 20.10.5+dfsg1 which is less than 18.2.0        â",
â", consider upgrading or you may experience issues. You will not be prompted  â",
â", again. You can manually upgrade by typing:                                 â",
â",   sudo apt upgrade docker docker-compose                                   â","
â",                           
So don't do it , for some reason it made my router choose a different IP address for the pi - took me awhile to figure out.
A few bugs with the IOTstack . I am using it to install Grafana, Influx, Mosquitto and Portainer
Back to Node Red
in command prompt   node-red-start
Go to Browser   http://127.0.0.1:1880/
to get Node Red webpage
Imported Jose flow again ( it didn't save from last time for some reason)

Gave up on figuring out how to make everything work in Node Red

Went on to install Create the ClassicMQTT container using IOTstack:
from this link https://github.com/ClassicDIY/ClassicMQTT/wiki/3.1-Raspberry-Pi-setup-using-IOTStack
which also tells how to install IOTstack
but since I already have it from above - I just wanted to install the MQTT for Classic.
Even though Graham has detailed instructions and screenshots - I messed up following a simple instruction and took a couple hours to figure out what I did wrong.
which was instead of "nano compose-override.yml"  like Graham says
I was doing " nano docker-compose.yml" which of course was wrong and would erase every time I tried to run it which I tried way too many times !
Anyway once I did it right it started right up and was able to use Graham's Classic Monitoring app on Android in MQTT mode.
Next I will see if I can adapt Jose's node red project to get the data from MQTT instead of from modbus.

That is enough for me for tonight ! 
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Graham

Larry, you can use Telegraph instead of node-red to subscribe to the ClassicMQTT messages and update influxDB directly
-Include Telegraph in the IOTStack build stack
- Grab the attached telegraf.conf, update the topics, username and password for the MQTT consumer (if your broker is not in a sibling container, also update the servers)
- ftp the telegraf.conf to ~/IOTstack/volumes/telegraf on your RPi

This will populate a database called mqtt_solar that you can setup as a dataSource in grafana

- Setup the mqtt_solar datasource in grafana
- import the attached grafana_dashboard.json

Graham.
Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s

ClassicCrazy

system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

ClassicCrazy

Quote from: Graham on December 01, 2021, 10:03:42 AM
Larry, you can use Telegraph instead of node-red to subscribe to the ClassicMQTT messages and update influxDB directly
-Include Telegraph in the IOTStack build stack
- Grab the attached telegraf.conf, update the topics, username and password for the MQTT consumer (if your broker is not in a sibling container, also update the servers)
- ftp the telegraf.conf to ~/IOTstack/volumes/telegraf on your RPi

This will populate a database called mqtt_solar that you can setup as a dataSource in grafana

- Setup the mqtt_solar datasource in grafana
- import the attached grafana_dashboard.json

Graham.

I think I have the telegraf .conf file installed
But setting up the Grafana datasource I get bad gateway error .
I haven't been able to figure this out yet .
Screenshots show my grafana 
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Graham

#4
you need to set your URL to http://influxdb:8086 in the datasource

(assuming your container name for influxdb is influxdb)

Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s

ClassicCrazy

#5
Thanks Graham - putting in Influxdb solved that gateway issue.
Now I get Influxdb database not found error.

Seems like IOTstack did some update  and messed itself or everything else up.
I am going to wipe it all out tonight and start over.

Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Graham

The telegraf.conf file is using the influxdb user/passwd that IOTStack creates by default, check the logs for telegraf (quick actions icon in portainer) to see if you're getting errors
Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s

ClassicCrazy

Quote from: Graham on December 05, 2021, 09:46:23 AM
The telegraf.conf file is using the influxdb user/passwd that IOTStack creates by default, check the logs for telegraf (quick actions icon in portainer) to see if you're getting errors

I wiped SD card and reloaded Raspberry Pi Bullseye and also IOTstack .
The last IOTStack update I did really messed things up. Or could be I messed things up.
I think there are lots of updates on both Bullseye and IOTstack being rolled out quickly so maybe second time fixed some previous errors.
And I found out I can make errors just as well in the daylight as in middle of night.
Latest one was I messed up the Telegraph .conf but spotted it in errors ( where you told me to check).
Anyway happy to report that now it is all working with your imported Graphana files.
Thanks for the help - I always learn a lot.
Here is screenshot ( super dark cold overcast day here so generator is running )
Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Graham

looks good,
The next step would be to convert the state of charge codes to English text. this is more difficult than you would expect in grafana.
You need to add the mariadb (mysql) container, create a lookup table in sql with the SOC codes and associated English text, then add a join statement to the SOC to do the lookup in grafana

something similar to this:
https://grafana.com/blog/2019/07/17/ask-us-anything-how-to-alias-dashboard-variables-in-grafana-in-sql/

Graham.
Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s

ClassicCrazy

Quote from: Graham on December 06, 2021, 08:04:04 AM
looks good,
The next step would be to convert the state of charge codes to English text. this is more difficult than you would expect in grafana.
You need to add the mariadb (mysql) container, create a lookup table in sql with the SOC codes and associated English text, then add a join statement to the SOC to do the lookup in grafana

something similar to this:
https://grafana.com/blog/2019/07/17/ask-us-anything-how-to-alias-dashboard-variables-in-grafana-in-sql/

Graham.

Figures nothing is too simple !
I have been looking a little bit under the covers at how you set the graphics up.
But have not spent too much time on it yet.
Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

ClassicCrazy

I figured out how to change the temperatures from Celsius to F
Just had to edit temp block and change this line
select BatTemperature   * 0.018 + 32   from mqtt_consumer
easy once I spent an hour figuring out the right place to put this simple fix in .
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

ClassicCrazy

Quote from: Graham on December 06, 2021, 08:04:04 AM
looks good,
The next step would be to convert the state of charge codes to English text. this is more difficult than you would expect in grafana.
You need to add the mariadb (mysql) container, create a lookup table in sql with the SOC codes and associated English text, then add a join statement to the SOC to do the lookup in grafana

something similar to this:
https://grafana.com/blog/2019/07/17/ask-us-anything-how-to-alias-dashboard-variables-in-grafana-in-sql/

Graham.

Graham
How do I get a list of the MQTT  identifiers ?
such as PV Current , BatTemp , etc  like in "select PVCurrent from mqtt_consumer"
I was thinking about adding my second Classic into config but don't know how Grafana would know which PVCurrent to get since they seem to be both identified the same way once in Telegraf ?
I may tackle the charge state sometime as you mentioned above.
Thanks for getting me started on this - works great. I  now monitor the Grafana page that is running on the Pi , but look at it on my Windows computer .
I will have to do one of the intro to Grafana tutorials so I fill in the many blanks of what I don't know about it
Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Graham

Quote from: ClassicCrazy on December 09, 2021, 07:23:51 PM
Graham
How do I get a list of the MQTT  identifiers ?
such as PV Current , BatTemp , etc  like in "select PVCurrent from mqtt_consumer"
I was thinking about adding my second Classic into config but don't know how Grafana would know which PVCurrent to get since they seem to be both identified the same way once in Telegraf ?
I may tackle the charge state sometime as you mentioned above.
Thanks for getting me started on this - works great. I  now monitor the Grafana page that is running on the Pi , but look at it on my Windows computer .
I will have to do one of the intro to Grafana tutorials so I fill in the many blanks of what I don't know about it
Larry

There is a payload example for "ClassicMQTT/classic/stat/readings" in the classicDIY/ClassicMQTT github wiki: https://github.com/ClassicDIY/ClassicMQTT/wiki/5.-MQTT-topics-and-message-structure
If you want to monitor a second classic:
1. Update your compose-override.yml file in IOTstack to include your second classic (see attached example)
2. Edit your telegraf.conf file using: sudo nano ~/IOTStackvolumes/telegraf/telegraf.conf
3. Add the topic for your second classic, then add topic_tag = "topic" (see attached example).
4. run ~/IOTStack/menu.sh,
5. Build Stack, the Enter to merge your compose-override.yml into the docker-compose.yml
6. Docker Commands, Start stack to run the containers, then exit IOTStack
7. You should now see classic_mqtt2 in portainer, check the logs to make sure it's connecting to your second classic
8. Restart the telegraf container
9. In Grafana, you now have to include a "where" clause in the select statements for each panel
ex: select BatVoltage FROM "mqtt_consumer" WHERE ("topic" = 'ClassicMQTT/MyShed/stat/readings')

You can use the grafana datasource explorer to query the series in mqtt_consumer, go to the influxdb datasource and press the explore button at the bottom of the page.

You can also use the influxdb container console, run the influx command (see attached screen shot)

Graham.
Off-Grid Island cottage, Lac Simon QC Canada
370 Ahrs @ 24V (4 8L16 batteries)
4 x 250watt panels on dual axis trackers http://tinyurl.com/hfpkgr5
Classic 150, Whizbang Jr.
Android Monitor: http://tinyurl.com/lomzq3s

ClassicCrazy

#13
Graham ,
I was able to get it all set up in Grafana for two Classics now.
Of course I made a few mistakes first but that really helps me learn !
I used Portainer to get into the container consoles and then your example helped me figure out how to go from there.
It was super overcast today so nothing exciting on the screenshot but I did get it running. Also figuring out now how to do things in Grafana
thanks again for the help.
(ooops forgot to save dashboard so have to do it all over again ! more practice )
Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

ClassicCrazy

#14
Quote from: Graham on December 06, 2021, 08:04:04 AM
looks good,
The next step would be to convert the state of charge codes to English text. this is more difficult than you would expect in grafana.
You need to add the mariadb (mysql) container, create a lookup table in sql with the SOC codes and associated English text, then add a join statement to the SOC to do the lookup in grafana

something similar to this:
https://grafana.com/blog/2019/07/17/ask-us-anything-how-to-alias-dashboard-variables-in-grafana-in-sql/

Graham.

I may have found easier way to do the State of Charge code to text.
There is a Value Mapping section .
https://grafana.com/docs/grafana/latest/panels/value-mappings/
Guess I will find out next time the sun comes out if it works or not.
Screen shot shows what I did .
attached is my latest version of Grafana Dashboard  supporting two Classics and I jazzed it up with colors.
Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable