Node-Red, ModBus, InfluxDB, Grafana on Raspberry Pi - Solar System Monitoring

Started by Steve_S, June 08, 2019, 05:31:42 PM

Previous topic - Next topic

Steve_S

Due to a suggestion by another Forum Member here ClassicCrazy I've copied this post out of another thread to create a new series thread here.  The software's used here are available for other platforms such as windows but I will not be addressing such, unless it's a part of "this adventure".

A WORD OF CAUTION in regards to Modbus
Modbus can Read & Write stored data in your Classic Controller & other devices as well.  This is a helpful tool to help optimize & tweak your systems.  That being said, it can be dangerous & destroy your equipment & batteries if misused and this cannot be understated !  Even the Midnite Docs & Others all state such plainly but some folks refuse to pay attention.  PLEASE DO ! 

To that end, my intention for this project is strictly for Monitoring, generating statistics and graphical representations of same, therefore I will be using only READ functions.  This will also present other general reference information for "display" but nothing that will change or modify settings on the controller. If anyone takes code from here, modifies it to make changes and modify a controller or anything else, it is at Your Responsibility, Risk & Expense !  Do NOT expect any manufacturer to cover such damage, it's considered self-inflicted, if you know what I mean

- End - Disclaimer

I'm just setting up my new Pi 3B+ for this purpose as well as monitoring a few other things.  I've taken this project a bit further as I am incorporating the following:

Hardware used:
- Raspberry Pi 3B+,
- DS3231 Real Time Clock module, !! This is important to keep current time as InfluxDB and many things require accurate time.
   Web access for such is not always guaranteed and it's best not to rely externally.
- Kingston Hyper-X 240GB SSD boot & data disk.  Using an external drive case with SATA to USB 2/3 interface. 

Software used:
Raspian OS
- Node-Red
- Node-Red Dashboard   https://flows.nodered.org/node/node-red-dashboard
- Node-Red-contrib-modbus 4.1.3  (includes Serial & TCP) https://flows.nodered.org/node/node-red-contrib-modbus
- InfluxDB Time Relational Database  https://www.influxdata.com/
- Node-Red-contrib-influxdb 0.2.2  https://flows.nodered.org/node/node-red-contrib-influxdb
- Grafana Analytics & Monitoring https://grafana.com/

I partitioned & formatted a 240GB SSD Sata Drive and set that as boot device because the DB will be storing a lot of data over time.  This makes my installation slightly different.

After installing everything with a Fresh Rasparian, node-red and the rest, I ran Peter Scargills "the script" and ensured that everything I wanted was selected  and let it update everything (this takes a while).  There were a few errors during that update so after a reboot, I restarted and reran it and all came out 100%.   The Script is GENIUS and strongly suggested.  Here it is:  https://tech.scargill.net/the-script/

There is no need for Apache / Lamp to act as a webserver as Node Red (dev & dashboard), Grafana communicate directly.  InfluxDB is a command-line interface.  IF you wanted to do this without Node-Red there is Modbus for Java & Python that would accommodate using regular webpages as in the other thread but that is IMO a more complicated approach.

This Tutorial / Overview got me started on this particular path.  Have a peek and look at some of his other videos which tie  many things together.  https://www.youtube.com/watch?v=JdV4x925au0&t=3s    Also, there are many other YouTube vid's that cover Solar & Node-Red and more... 

I may actually do up an article on setting up a Pi for this purpose including how to setup an SSD or other HDD as the boot device as there is some out of date info that can lead down a wrong path.  Of course that will likely take some time as it's mid busy season now that summer is attempting to start !  (well it seems that way at this moment).

BTW:  I have been playing with the flows posted in here...  some good potential in all of this and room to really expand on.  I will be posting flows as I work my stuff out.

A note to ponder on:  I'm using a combo of Node-Red Dashboard because not only can that be used to present the data in Real Time (system status etc) but it also has facility send setting changes (with extreme caution) to the Modbus device (using the Modbus lib I posted).  The Grafana and InfluxDB is for the longer term view of performance & generation and allowing to generate a nice view that results from the data collected.   Ultimately, I would like to have my Classic-200, the Samlex 4024 Inverter/Charger and my Weather Station all  plugged into the Pi and collecting data. 

--EDIT:
Concerning Network Security, Application security etc.
This project is dealing only with the flows & putting this together to make it work.  I won't address security in regards to configuring routers, firewalls an what not.  That's too much of a diversion from the intent.

Hope that Helps,  Good Luck.
Steve

Steve_S

OK, here we go with Try #1

This is Node-Red with Node-Red-contrib-modbus 4.1.3 and Node-Red Dashboard only.  No data storage or anything at this time.

Attached are two Flows, the first is the Modbus TCPIP to the Midnite Classic which retrieves the data and formats it. 
The second is the Node-Red Dashboard which receives the data and presents it on one page. 

See the attached Dashboard & two Flow images.  Quite basic, simple but a fair start considering I'm new to Node-Red...   not new to programming etc... though. 

This is far from optimal code, it's sort of "rough & ready" but worth experimenting with.  The registers and setup is a result from my downloaded Registers from my Classic Controller (using the Midnite App) and the Modbus Documents provided by Midnite.  As a result, I attempted to use similar naming conventions for the variables to keep it consistent and easier to troubleshoot.

I was going to paste the code in for the Flows BUT they exceed the limit of what's allowed on the board.  So I'm attaching a ZIP file instead Classic-Getter_w-Dashboard-V1-Basic.zip  which includes the combined Two Flows for the Getter & Dashboard in one JSON file, the Getter Flow JSON & the Dashboard Flow by themselves.

Here are the Variables as being used:
"BattVolts":26.7,
"InputVolts":120.6,
"OutputAmps":8.2,
"kWhToday":2.9,
"WattsInNow":219,
"InputPvAmps":2,
"InputPvVolts":131.8,
"HighVoltInToday":187.6,
"AhrsToday":108,
"LifekWhrs":81.1,
"LifetAHrs":2939,
"BattTemp":24.4,
"FetTemp":47.5,
"PcbTemp":48.4}


Good Luck & Enjoy
Steve

NOTE:  Ensure you have nothing else communicating with your Classic Controller, it only handles one connection at a time.  With Node-red, ensure you have no other flows attempting to talk to the controller, as well if you have any "unused" Nodes in Node-red, they can still cause havoc with your attempts.


ClassicCrazy

Thanks Steve - I am a bit too busy now but will load up you code in the coming weeks and give it a try . 
Thanks for your detailed info

Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Steve_S

The above V0.01 is pretty basic stuff... I used what was "commonly" used in the other scripts etc as a starting point to get things to work, pull in the data and generate some reasonable dashboard output.   It's quite rough & ready as I was just working at getting stuff out of the classic for a start.

I've now got a node writing to the InfluxDB (man that is slick !) and put together a test Grafana panel.   Having done that now, I've come to realize some things are going to have to be changed a bit to accommodate what I want to get out of this.

There is a wee bugaboo in this though.   All of this is of course FREE as everything is opensource but it depends on having assorted things installed to make it work and as such this is not as "user friendly" (aka plug & play install).  One option would be to put together a "Package" and bundle it up as a prebuilt Image to burn to SD Card or...   Still not as "user friendly" as I would like.  Nothing beats clicking on "Install" and having a program set itself up and only need a few questions answered and voila, your using the software.

Ideally, I would like to make a Cross Platform OpenSource (free) software package that talks not only to Midnite Products but other Modbus / other solar devices like Inverters etc...  Sadly, not everyone is Open about their Modbus specs, in fact I have to sign an NDA to get the Modbus specs for my new Samlex Inverter...  That means I cannot expose the Modbus registers/ports to users... which means encrypting that somehow...

I'm NOT interested in getting back into C# Programming and now with new Visual Studio 2019, I'd have some serious catch up to do and the cross platform portability issue is one of my big concerns.  Cross Platform & free is important to me...   Anything to help people get off the Grid Power Addiction for a cleaner & better future for all of our kids, grandkids etc.   Just can't leave them the same inheritance we got, because now we know better.

ClassicCrazy

I guess if you get it all going and want to share it - you could delete any personal information and then make an image of the SD card for Raspberry Pi  and put it on a cloud source to download.  That is how Andries Spies did it I think - I just download the one he had and used Balena Etcher or Windisk32 to write the image to SD card - put that in Raspberry Pi and it all just worked .  The only thing I did before I put it in the Pi was edit the folder for wifi to put my own router and password in and then it connected right up to my network.  I recently made a copy of a raspberry Pi Zero W here that had Weewx weather station running and I put it in another Pi Zero and it all worked .

Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Steve_S

Well I do want to eventually "share" what I develop so that folks can integrate their various Solar Bits & stuff together to help them improve & tweak their systems to get most bang for buck out of them.  The image is likely the best "overall" and that isn't hard BUT I won't be doing that with every addition to this, only when it's getting to the point of something real.  At this stage, this is "playing" with the software.  I've downloaded a lot of Andreas' videos from YT and used his examples & tut's for some of this, even have his Grafana image but I went slightly different.  Downloaded many others as well.  Much easier for me, as I'm on Satellite Link and when I turn it off, no web.

To that end, I now have Node-Red writing to Influx DB (boy that is slick ! ) and Granafa talking to it and that is amazing too BUT this is a bit of a learning curve.. not as easy as the videos & texts say it is...  With further tinkering on the Modbus end, I can now pull 100 registers in a shot and organize that to usable info.  Although, this is not entirely without other issues, so I'm looking at how to manage the register data retrieval as needed / wanted.  The timing, timeouts etc are somewhat "tricky" and require finesse to get that tweaked.   I wonder if any Midnite Inc. Software guys are reading this as it would be great to get their recommended timing specs etc.

On the Samlex Inverter front, we reached an agreement, understanding and with an NDA (non-disclosure agreement) for development I have their info & support, PLUS they're likely gonna be peeking at this thread !  ;-)   This is great as there is valuable data in there that can be tracked for optimizing & performance monitoring.

RS323 / RS485 Serial to Ethernet
The Samlex uses RS485 so instead of using a Raspberry Pi "HAT" for the interface, I chose to go with an external converter to convert Ethernet which will also provide me with another debugging tool as it has it's own IP interface.  Keeping prices reasonable & with a recommend from Samlex Technical I ordered this one from Amazon

USR Usr-Tcp232-410S RS232 / RS485 Serial to Ethernet Adapter/IP Device Server Ethernet Converter Support DHCP/DNS
https://www.amazon.com/USR-Usr-Tcp232-410S-Ethernet-Adapter-Converter/dp/B07C1TC165/ref=sr_1_1_sspa?keywords=USR&qid=1560464781&s=gateway&sr=8-1-spons&psc=1   Hopefully that will arrive by end of next week.

BTW:  An interesting aside for anyone reading or interested...  Most "On Grid" systems comply to SUNSPEC MODBUS making them more easy to communicate with, across brands...   Midnite isn't on there, didn't expect it as they are not in that segment as such, yet ?  but with B17 and more coming, who knows...
QuoteSunSpec Certified Products â€" SunSpec Modbus
SunSpec Alliance certifies DER system components and software applications for compliance to official and de facto communication standards. If you are interested in having your products become SunSpec Certified please contact certification@sunspec.org

ClassicCrazy

Not sure your Classic setup , but if you have serial access available it might work better than getting of of modbus via IP.  I had to give it up when I added second Classic and set up Follow Me since it doesn't seem to send out regular modbus any longer - not sure if the top MNGP port would work even if bottom two are in Follow Me.
Take a look at the link I made on how I hooked up to Classic for Arduino.  It might also be handy to get data out of other devices in similar way .
http://midniteftp.com/forum/index.php?topic=4046.0

Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Steve_S

My setup is fairly straight forward (to me anyways LOL). 


  • Powerhouse / Pumphouse is 75' from cabin.  Solar Rack is 40' from Powerhouse.
  • 2 - Cat-6 Ethernet lines, 1 satellite line run in 1 underground Insulated conduit to cabin.
  • Power & Water through another insulated conduit ( 3/4" Water, 1x AWG-6/3 & 1 AWG 8/3 NMWU

The catch here is that I only want the Sat up when I need it (60W power sucker) for internet.  The Pi, Router can both run in the powerhouse off the 12VDC tap from my battery bank.  Ideally, I'd like to have the Sat Modem in there too bu I'd have to get some kind of remote controlled plug outlet for that, so I can flick it off when not wanted.   NB:  Sat is Internet Access not one of those TV by Sat jobbies.

I haven't any issue with the Modbus running though IP, although there is a bit of timing tweaking to do yet, it's just annoyance stuff ATM, not errors of concern.   TCPIP is certainly a better option for me.  I had considered possibly using WIFI but that idea came to an abrupt end because not only the RFI in the Powerhouse BUT it's sitting next to a 50 ton magnetic erratic boulder.  (I'm on top of a granite ridge which is what is left of a mountain range the last ice age levelled, as a result there are "erratics" everywhere)   I've even got Lodestone in spots that wipes out cell phone access and makes compasses spin.

I was thinking on getting an RS485 "Hat" for the pi and going that route BUT by using the RS232/RS485 converter I can plug other things in there down the road.

Phew, chewed down all the Samlex Data into the wee hours last night… geez, that's a lot to chew on but good stuff.  Boy I love Hex Math !  NOT...   Flashbacks to programming in assembler, goodness me.

ClassicCrazy

Curious about your 12v tap off the batteries .  Do you mean that you are using a DC to DC converter to get 24v to 12v ? Or are you just tapping off of half the batteries to get the 12v ?  The second option isn't the best for batteries. 

Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Steve_S

I used a proper DC-DC converter which provides 12V / 20A which is also fused properly.  If you think my posts are detailed, well I was an IT Systems Engineer for my main career and started with Mainframe Engineering in the 80's....  I do not like half-assed anything.

Attached you will see a photo of my power room (while being assembled) and my diagram for THAT revision.  That is before the Samlex and related upgrade gear, which I have to get around to updating sometime soon.  Maybe that will help a bit to explain it.  Pictures = 1000 words. LOL

ClassicCrazy

system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

dgd

That’s a neat wall setup with the MNDC, but like CC mentioned there does not appear to be a WBjr battery current sensor attached to the Deltec shunt. You are therefore missing out on a whole set of  modbus register values that would provide very useful battery management info as well as the ability to terminate ABS charging based on ending amps setting.
The charge status screen you included above with the meters and tiny graphs, is that configurable in that different meter styles can be used and size adjusted? Can larger more detailed versions of the graphs be included? Perhaps on additional screen displays?
Look forward to loading a working monitor system on to my Pi3 currently in cupboard acting as a dust magnet.
Dgd
Classic 250, 150,  20 140w, 6 250w PVs, 2Kw turbine, MN ac Clipper, Epanel/MNdc, Trace SW3024E (1997), Century 1050Ah 24V FLA (1999). Arduino power monitoring and web server.  Off grid since 4/2000
West Auckland, New Zealand

Steve_S

As I mentioned those are Diagrams / Pictures of my previous system with the Yiyen APC-3024 Inverter. 

The Revision has a WizBangJr (which I had to buy, even though I sent in the card to get a Free One), there is also an Midnite Battery Monitor (for at a glance status, at the battery bank).   Add to that, a 40A AC Breaker In from the Gen-L5:30 line and another Single Pole 45A between the Samlex & AC Panel.  I am awaiting 2 more items (in transit) before I completely assemble my revised / updated system at which time I will diagram it & photograph it.   I document everything !   The entire build here, which is very unique Net Positive, semi-self regulating home. 

Quick Home Spec: built with 5-1/2 Thick HD EPS-II foam in the walls, 7-1/2" of PolyISO in cathedral ceilings, with a "Texas Cool Roof" providing 2" air space channels to ridge vent and Rainscreen method installed Live Edge Cedar siding with air channels feeding to soffit for draft.  The foundation is a Frost Protected Slab with 4" of HD-XPS underneath and 2' out the sides, containing 7/8" Radiant Pex.  The thermal control from the slab maintains & regulates internal temps year round.   My Powerhouse / Pumphouse are identically built as well, so very tight & efficient.   ALL Foams were supplied from Commercial Roofing companies as this material is generally not available to the Retail Market and what is available @ retail is at Maximum Market Bears Pricing which is 150% Ludicrous !.

ClassicCrazy

Your home sounds very efficient.
How does your control deal with cooling the home - by just not sending in warm water ?  How do you heat the water for slab ?

I just watched the new Andries Spies video - I am happy to say I actually knew how to do one of the things he mentioned - hurray ! That was how to set up headless for wifi and ssh .
But his tip about putting on the button for shutdown is something I will put on because I have had on occasion I couldn't get into the pi via ssh because something got locked up and I had to resort to pulling the plug . The other tip was for making a change for Grafana which I wonder if would have helped with the other problems that caused the pi to lock up.  But since I switched to a better SD card in the Pi I have not had many problems .

Larry
system 1
Classic 150 , 5s3p  Kyocera 135watt , 12s Soneil 2v 540amp lead crystal for 24v pack , Outback 3524 inverter
system 2
 5s 135w Kyocero , 3s3p 270w Kyocera  to Classic 150 ,   8s Kyocera 225w to Hawkes Bay Jakiper 48v 15kwh LiFePO4 , Outback VFX 3648 inverter
system 3
KID / Brat portable

Steve_S

I saw bottle necking when I tried using an SD Card but after switching to using an SSD drive as boot & storage, so much quicker, especially when writing to influx which peaks the CPU to 20% now for a blip while it writes.

I built my own place myself, only contracted the concrete pour, well drilling (260') and for the roof finishing.  My Radiant System comes from "The Radiant Company" http://www.radiantcompany.com/ which was a Very Wise move for me and they make an excellent packages up for folks to use, they do the in-house design & setup for you at no cost.   The radiant heat is provided by a dedicated Takagi TH3M On Demand Heater (I have Cryotek-100 anti-freeze in the system) {YES the Non-Toxic stuff !}  The slab being thicker than normal (6" @ centre, 10" on sides) and being completely insulated, it is a wonder regulator.  IE:  During heat wave last summer, we hit 42C up here on the ridge, inside it was 27C and I do NOT own an Air Conditioner or a fan, the floor reached 23c at the hottest point).   In winter, -35C outside, heat loss with No Heat being generated is 4C in 24 hours.  My Hot Water comes from another On-Demand system Should point out that my Windows & Doors are also installed using Bailey Boxes to isolate them.  I have no wires or plumbing inside ANY external wall and therefore nothing to reduce the efficiency.  No, you cannot see any wires & pipes inside either.  By the way, the On-Demand Heater for the Radiant System only runs a total of 1 hour 15 minutes total in 24 hours (broken up in 3 runs typically) when it is -20C outside.

Thermal Mass if used properly is the primary heat / cool regulator in my home, as such, in summer the slab drops to roughly 17-19C and remains there unless I add heat to it.  As such it prevents heat buildup and never an issue with condensation at all.

Air Management is done by a DIY Air Exchanger Recovery system I built myself, it is based off of this design / idea.  https://www.wildsnow.com/17884/how-to-build-air-cross-flow-heat-exchanger-budget/  but with mods to suit my use case and needs.  I used some different materials and finished the external side "nicely" by using an IPEX System 636 Concentric Gas Vent
like this (which matches my On-Demand Heater Vent)

Geez, this thread veered didn't it...  Oh Well, hopefully others are getting more info they can use for themselves too.   Conservation is far cheaper & more efficient than generation.... ;-)  All part f a Holistic Design view... start from the ground up !

Now, off to clear some bush while it ain't raining !