CCDM - Charge Controller Data Manager

[Fort Wisers]

----------------------------------------------------------------------------------------------------------------------------------------------------
NOTE:
Although this thread is placed in the raspberry pi section, CCDM can work on any device running Windows, MacOS and many distributions of Linux (including raspbian)
----------------------------------------------------------------------------------------------------------------------------------------------------


Hey everyone, I hope everyone is keeping well and to our friends south of the border please have a Happy Thanksgiving!

Over the spring I threw together a quick python based application then alpha tested it over the summer, as in the subject, I have been calling it CCDM (Charge Controller Data Manager).
In this thread I wanted to:
1. Show some screen shots of the progress as it stands today
2. See how much interest there is out there for an app like this
3. Possibly gather ideas for further development down the road

In our case we're running a Rasp Pi (3B+) with CCDM on it polling our Classic 200, 24/7 at our cabin up north. I have setup Real VNC on the Pi so I can check in on things at any given moment and update files as development continues. I also get my daily emails from CCDM for a general sense of how the day went. We have the Pi mounted on the back of our TV (in a little 3D printed case) and have the Pi also feeding an HDMI input to the TV. So, when we're at the cabin, I can see the application running by simply turning on the TV and selecting the appropriate HDMI input (our kids have dubbed it dad's solar TV channel lol).

My plan over the remainder of this fall and into winter is to develop it a little further, create at least a rudimentary document package and then post a beta release publicly on GitHub with an open source lic. Time has been a precious commodity on this end lately though so, outside of some very minor tweaks, I haven't had a moment to touch the code in a couple of months.

Anyhow, here we go:

Overview Tab - the overview tab gives you just that, a general overview of what your system is doing real time (1/2 second update rate).
Note: It's been very cloudy this fall with very little sun, hence the reason I got a screen shot with a poor example of our solar system's output.
This very low system output seen below leads to the low CC eff values that you see in this screen shot, don't worry MidNite fans, our classic generally runs around 92-94% eff.



Misc Tab - in addition to the connection information dialog (as also seen on the Overview screen) this tab displays some peak values and events that occurred during the day (reset at midnight each evening) and also gives a break out of Info Flags. More to come on this screen, hence the large blank section on the RH side:



Config Tab - as with the previous two tabs the top left is our general connection information dialog. The other dialog on this screen shows the end user some general configuration data on how they have the CC configured. Once again I have some plans on what dialog will be filling in the blank section on the RH side of this screen:



There is also an About tab, this simply displays versioning information, lic information and platforms the code has been tested on.
On that final note, so far I've run it successfully on Win10, Linux Mint 17, and Raspian Stretch (rasp pi). Since it's python based there should be no issues running it on any OS that can run a python interpreter, so Android (as is) or iOS (running pythonista or similar) SHOULD be able to run the code. I say should in caps as I have not tested this yet.

Some other features and notes about CCDM worth noting:
- At 11:59pm each day the application generates a log file summarizing a lot of the data collected (peaks, events, kw/H and Ahr for the day etc etc), this log file is date stamped and
   saved as a plain text file (one file per day) in a folder local to the machine that CCDM is running on
- The same data wrapped up in the log file can also be emailed to the end user at the same time. I will note at this point however that I have more development to do on this end as I currently
   do not have sensitive email account information (ie password) encrypted (currently stored in a plain text file only) so I would not feel comfortable releasing this as a final product without some
   serious disclaimers. But the general functionality is working and proven, its simply a matter of finishing off some bits and pieces to make it secure.
- Currently, configuration of the application (setting things like what IP address your Midnite CC is at) are all wrapped up in a file that can be easily edited with any text editor. In the future my
   plan is to move this out to the UI so no file editing will be required to simply "get it off the ground".
- The code is also preliminarily complete to talk to two other CC manufacturers but has not been alpha tested, I'm trying to get my hands on some hardware to complete this.
- Note the "Auto Scroll" check box on the top RH side of every screen, if the user checks this, the application will then scroll through each screen tab at a set interval (adjustable). Handy for
   watching my "Solar TV channel" where I don't have a mouse or keyboard to flip between tabs manually.
- I've done all development so far in notepad++ and idle, both free and simple environments / editors

Plans for the near future (near being a relative term based on how much time I can devote to development in the next coming months!):
- finish off dialogs on the Misc and Config tabs
- get CCDM configuration functionality onto the UI to avoid the need to edit any config files manually
- alpha test the two other CC oem products
- re-develop the UI in HTML5 (the current UI is done in tkinter (a package currently built into python, I chose to use tkinter as a simple method to prove out data collection and functionality))
- wrap-up email sensitive information into a DB for security
- collect the daily log file data into monthly / quarterly / yearly reports
- develop a doc package for end user use
- get it publicly onto GitHib with an open source license

Lastly, some of my inspiration for starting development of CCDM was as a result of using Graham's Classic Monitor Android App which we used as our original method to watch our solar.
I really like the work Graham has put into his app but we simply needed something not bound to one OS, hence CCDM.
Many thanks to Graham for all his hard work in developing Classic Monitor and to Midnite for a great CC product.
And thanks to anyone who actually took the time and interest to read this post in it's entirety.

I think that's about it.....
As with the development of CCDM itself, I won't have a ton of time to check out comments on this forum, but feel free to leave your thoughts and I'll check back in a few days to see what level of interest there is (if any). Otherwise, I'll post a quick note once I finally have it on GitHub and others can do as they feel fit with it!

Cheers, take care everyone:
Brent

[boB]

OMG, Brent !  This is beautiful !

That setting panel is exactly the kind of thing I was wanting to make, personally !

Wonderful job !   I'm sure you will get rave reviews of this !

[ClassicCrazy]

Brent -
Your project looks very nice.
Is it going to be able to write to the Classic to change settings ?
Have you kept up on the other Classic projects that Graham and Matt have developed in the past year ?
One of them uses the Pi to create MQTT data which then lets any other program use that at the same time - so it gets around the limit of only one Classic IP modbus connection at a time.
Graham also has an ESP32 that can connect to serial port and then create another Classic modbus IP port so that also lets more than one device connect at the same time ( as well as lets monitoring happen for the Classic models that have no IP networkings ) .
In case you haven't kept up on those other projects they are here
https://github.com/ClassicDIY

Larry 

[Fort Wisers]

boB -> Thanks for the kind words of encouragement.....if I ever get this thing on GitHub we'll see how reception goes.

Larry -> My motivation for starting this project was multifaceted, one being the ability to write to the CC.
At some point it was time for a firmware upgrade on our Classic, so I did as such.
This was right around the same time I was toying with the idea to create an application that could work on multiple OSs.
While going through menu after menu hand writing parameters on a piece of paper as backup (before doing the firmware upgrade) I thought it would be handy to have a Backup / restore feature built into the application, as well as the ability to adjust values of some of the more commonly used parameters.
My first step was to get something up and running, in a programming environment that could be run on Win/Linux/OSx.
I did, during development, trial write to one register in the CC without issue.
However, giving the application the ability to write parameters comes with some extra overhead of thinking on how to somewhat protect an end user (and CC) from inadvertently entering potentially dangerous values. I have a few ideas on this, just not the time to actually code anything.

So I haven't taken this part any further since my primary focus, for now, has been on reading, displaying, logging and emailing information.

To summarize, the application is currently read only but I would like to add a backup / restore feature as well as some simple ability to adjust some values in the CC from the application.

I have not checked out the work Graham and Matt have being doing (other than using Graham's Classic Monitor sometime ago).
I had brief conversation with Matt last year in regards to remote connections, and that was about the last time I checked out the forum before starting this thread.
Will check out their work as time permits, thanks for the link!

Take care everyone:
Brent

[UpNorthMan]

Brent,
The work you've done for remote monitoring looks great! Very easy to navigate and understand.It looks to keep you well informed of the status of your remote cabin. I know your time is valuable,so thanks for sharing it with all of us.
I have a couple questions,

1) Have you designed your code so it will support multiple CC?

2) From our Kid experts, is the information that Brent displayed, available to pull from a Kid CC?
I believe there are a few Kid owners that would love this elegant way of displaying our remote sites.

Ed

[ClassicCrazy]

Brent
From what you said about having to write down all the settings between firmware updates -  you know that you can save and restore them using the Midnite Local Status app ?
That makes life a lot easier for sure when doing updates !

Larry

[ClassicCrazy]

Brent,
The work you've done for remote monitoring looks great! Very easy to navigate and understand.It looks to keep you well informed of the status of your remote cabin. I know your time is valuable,so thanks for sharing it with all of us.
I have a couple questions,

1) Have you designed your code so it will support multiple CC?

2) From our Kid experts, is the information that Brent displayed, available to pull from a Kid CC?
I believe there are a few Kid owners that would love this elegant way of displaying our remote sites.

Ed

Ed,
The Kid doesn't have any modbus like the Classic does. It only spits out a very limited amount of data on serial.  There probably is a way to have a program like Brents display what it does have if some smart programmer put their mind to it. But the Kid needs a wired connection where the Classic can just pull the data off IP network. So it isn't nearly as easy to do like for the Classic.
Ricardo had his Kid displaying the data via Node Red and if you look in the Open Source section of the forums under Node Red I think you will find what he had done - but it was a few years ago. There also was a Kid firmware update that broke the data output - I think it has been resolved but I haven't kept up on it. Midnite is not putting any more development into the Kid so what there is now is pretty much it for monitoring.

Larry

[UpNorthMan]

Thanks Larry for info. I will investigate Ricardos' post. I'll talk to a friend and see what he can come up with.
Ed

[Fort Wisers]

Hi Everyone......
To answer a couple questions asked.

1. The code is about 25% complete to handle up to 4 independent CCs simultaneously (IE there is a plan in place for this, just not executed).
When complete it will work like this:
If you configured CCDM to collect data from 2 CCs (for example) you'd simply get a second set of Overview, Misc and Config tabs and they will be labeled
Overview - 1, Misc - 1 and Configuration - 1 for CC1 and the all the same Overview - 2, Misc - 2 and Configuration - 2 for CC2.


2. Adding a serial interface to talk with the KID CC (or other CCs that offer a serial I/F only) would be quite easy, maybe I'll add this to the to-do list.
I have not looked at KID docs though at all to see what is available to gather in terms of data points.

Cheers:
Brent

[ClassicCrazy]

Thanks Larry for info. I will investigate Ricardos' post. I'll talk to a friend and see what he can come up with.
Ed

I believe I remember Graham mentioning that it would be possible to modify his code to be able to bring the Kid data in . I don't remember exactly which project it was though.
Once the data is in the Raspberry Pi and MQTT you can pretty much do whatever you want with it. I am only a tester and not a programmer so don't know all the details.
Take look here
https://github.com/ClassicDIY
Larry

[UpNorthMan]

Brent,
I'm glad you had the for thought for multiple units in your design. Many Midnite owners will be interested in your product as it progresses tword a finished product. Please keep us updated!
Ed

[Robin]

We are shipping the first high voltage controllers this week. They are going to MidNite employees on the East coast and midwest. The first ones are only 75 amps output with a 600V input. You guys will have fun writing software to go with all the new stuff. It is Canbus! We will also have a relatively inexpensive combox that receives canbus, and modbus over the RS232 port. It also has a third party canbus port for other manufacturers batteries or whatever.
Things are starting to work pretty well, so we will keep you informed as things progress.

[Fort Wisers]

We are shipping the first high voltage controllers this week. They are going to MidNite employees on the East coast and midwest. The first ones are only 75 amps output with a 600V input. You guys will have fun writing software to go with all the new stuff. It is Canbus! We will also have a relatively inexpensive combox that receives canbus, and modbus over the RS232 port. It also has a third party canbus port for other manufacturers batteries or whatever.
Things are starting to work pretty well, so we will keep you informed as things progress.

Hi Robin, sounds interesting, which CC is this? Do you have a published document laying out the CAN mail boxing scheme?

Sorry everyone for the lack of responses.........I my just publish CCDM as is (on Github) and list out a set of feature additions to be implemented at a later date.
I can pick away at it slowly as winter roles in and push updates after testing.

[Urayoan]

awesome!

[boB]

I think that our CAN protocol is mostly done but it is still somewhat of a work in progress until we have more experience with it and especially our beta users operating the system.

I'm not sure about future publishing ?   I am fairly sure users will be able to interface with the newer system but I don't know yet exactly what that will include.