log files/csv files

[dgd]

I have spent some time thinking about log files for data from the Classic and KID and would be interested in what, if anything, other Classic users are storing for either immediate use or later processing into reports and graph displays.
ALthough I can get about 1.3 second repeating readings via rs232 of Classic Modbus registers I cant think of any good reason to store every data set.
For a basic text report I have captured snapshots every 10 minutes then list these to cover about 2.5 hours worth on one screen, displaying in latest at top to oldest entry at bottom. Had thought about 10 second snaps then averaging for 10 mins but that didnt seem to add anything useful, just uses lots of storage.
The most annoying anomoly is those MPPT sweeps to low watts as these get captured too often.

The arduino project log files to SD card proved almost useless as the SD interface is just so very slow. Instead just use a set of memory arrays and keep overwriting 50 entries maintaining a start pointer.
Looking at two modifications to this scheme, when no change to data during night,
dont store snaps just a start time to end time special record, this ignores decreasing battery voltage, should minimise storage over non activity periods..
ALso when system sleeping then do something with writing to SD storage.
Had also considered compression but hardly seems worth it for a limited set or variables, for same reason not using complex storage structures  eg databse.

dgd

[zoneblue]

Yeah im also not sure. 1s data is held for all registers, and bzip compressed at midnite, (maybe a few megabytes per day), but the sql db only contains 60s agregates and daily agregates. Even 60 is a lot of data, for a flot based graph thats 1440 samples, about 20K of json, and a second or so of (arm) server time to extract and deliver. Most commercial monitoring resamples all this to 5 or more minute intervals for obvious reasons, and yet i consider the devil can sometimes be in the detail and like to see whats actually happening.

[dgd]

I'm not actually sure what data to store (semi-)permanently on SD or HDD.
Already I'm hypnotized just watching the live-data gauges  display of what the Classic is doing but now intersperced with looking at the 20 by 10 minute log entries to see what pattern there has been over the last 3 hours. Too often just to see if the next 10 minute record has appeared yet.
Thinking about these 10 minute records, I have just opted for a snapshot of Classic data every 600,000 millis which should be 10 minutes (but the Classic clock is pushing forward 5 extra seconds per day - can the Classic's clock be tuned?).
Snapshotting versus averaging 10 minutes worth of 1.3 second records seems to make little difference except when an event occurs as the snapshot is taken (like boiling the electric jug for a nice cuppa) and Battery Amps sinks to -72.6.
All the extra storage and writing to it plus the sums every 10 minutes to get the average record has not convinced me its worth the effort.

But graphing for a days worth, or several days worth of data into a pretty graph will probably need more effort.
I really cannot think of what I would use the graphs for that would be worth all the data gathering, storing, processing and graph generation effort.
Maybe its just me not seeing something obvious but what apart from pretty graphs, knowing there is order in the RE universe
(ie the Sun keeps rolling across the sky, wind keeps blowing etc) and I had enough power yesterday to make the pot of coffee that I made and enjoyed, is the real purpose of these historical or near past event graphs ?

dgd

[zoneblue]

Yeah im not too keen on those monitering systems that agressively average, enphase  is a good example. Sure it looks a nice smooth curve, but theres a lot hidden by it. So like you i snapshot for the 60s agregate. That way the fridge cycling still looks like a square wave not a rounded hump.

Personally i like the current day graph. From it you can tell exactly when absorb started, ended, or how close it was to float if it never made it, what sort of load stressors might have occured that day and when and how much.  Im about over the classic's SOC, its just about always wrong, given that my classic reboots so often. Time to roll a blackbox SOC data series i guess.

Theres some indicators im working on. Rest voltage, i do this by specifying the time of day just prior to day loads starting. Then taking the highest voltage in the previous 30 minutes.

Internal resistance, still thinking this one through, but something like delta V, delta I for some period. Could be interesting to watch as the bank ages.

[dgd]

So for me it looks like a current days graph and maybe a previous days graph would be all thats ever needed.
Worst case, one second data storage record for two days is 144,000 records and allowing for 64 byte per record thats near 10megs total storage
With serial port data timing about 1.2  seconds and doing something smarter in non power production periods this could be halved. Certainly for inactive PV input at night there  is little point in storing records
With wind the night rule would not apply but it should be possible to not store data when wind power is nothing.
One problem I see is dealing with data logging to SD card and cpu cycles consumed by moving this data.
For the Arduino this would have to be memory buffered as much as possible, not an issue with the DUE and its 96k of ram, but  for Mega with 8k its a problem. Maybe Sata interface to HDD on Cubie woud be ok.
I know some HighCharts can deal with this amount of data but need to look at other charting options.

Im about over the classic's SOC, its just about always wrong, given that my classic reboots so often   

I sorta got the SOC with my elderly bank just about right but it still surprises me to see a 85% soc and the Classic goes to Absorb, runs 2 hours with soc to 94% then transitions to float and the soc jumps to 100%
Seems to only happen when about 1700 to 2000 watts input occurs for a couple of hours.
With lower watts, slower to absorb it gets to float just as soc 99% 

So 'reboots so often' means what? once a week, day or more often?
from reading other posts I thought these issues were now minimal or gone. I suppose unplanned reboots make logging very variable or at least with noticeable gaps.
I suppose I'm lucky in not having a reboot for over 10 months now but it probably because I do not use the Classic's ethernet interface for any data transfers, neither local app nor Mymidnite.
Data to/from serial port seems rock steady and dependable.
Maybe getting a serial modbus interface from Cubie to Classic will be the way to go, I noticed that libmodbus is included in most Linux distributions and it supports serial and ethernet modbus

dgd

[zoneblue]

With serial port data timing about 1.2  seconds and doing something smarter in non power production periods this could be halved.

Whats the limit there, serial speed?

Certainly for inactive PV input at night there  is little point in storing records

Yes i started out with this approach, but with the WBJr came information about battery current and loads, and these occur 24 hours. Hence i moved everything to 24 hrs. Load data is neat because it means everything is now in one place. However, the caveat is that the classic records and stores WBJr and classic amps in quite different ways. Subtracting one from the other sometimes yeilds negative loads but this is only during fast changing solar conditions. There is also an outstanding bug where classic amps is wrong by several amps at  certain times. Both those things aside, load watts is still fantastic to know.

One problem I see is dealing with data logging to SD card and cpu cycles consumed by moving this data.
For the Arduino this would have to be memory buffered as much as possible, not an issue with the DUE and its 96k of ram, but  for Mega with 8k its a problem. Maybe Sata interface to HDD on Cubie woud be ok.

I can see how how atmel will require special consideration there.

I know some HighCharts can deal with this amount of data but need to look at other charting options.

Flot says something to the effect of if you have more samples than pixels you need to rethink. However a couple of weekends ago i did add flot to theblackbox. 1440 samples renders quite fine even on low power devices like netbooks. Suprising good. And of course load average on the box lowered noticeably. I found flot to be relatively easy to configure, with only one or two small anoyances.

I sorta got the SOC with my elderly bank just about right but it still surprises me to see a 85% soc and the Classic goes to Absorb, runs 2 hours with soc to 94% then transitions to float and the soc jumps to 100%
Seems to only happen when about 1700 to 2000 watts input occurs for a couple of hours.
With lower watts, slower to absorb it gets to float just as soc 99% 

As far as SOC algos go bob said that it was relatively basic, accounting for the two factors ah effciency and temp compensation. Im good with that as its a 'near enough' datapoint for a thing that is probably never going to be right no matter how complex the algo. Such are batteries, over reliance on soc data, SG, knowing your bank etc. Anyway FWIW my agm bank and the classic WBJr defaults, (when it is working) the end absord SOC is usually pretty close. Never seen more than a percent jump there. Maybe AGM absorb being more efficient?

So 'reboots so often' means what? once a week, day or more often?
from reading other posts I thought these issues were now minimal or gone. I suppose unplanned reboots make logging very variable or at least with noticeable gaps.

Its eratic, it varies from once a fortnite to twice a day. The average is about 3 days. As you rightly say, i am 100% convinced that this is ethernet related. This is because the reboots only started when i started using a single open connection, and mirrors what happens when people intensively use local app which also holds a single connection.

Ethernet v serial. When i evaluated this earlier on, my decision to persist with ethernet was based on the effect on followme. It seemed to be a bad idea to design a monitoring system that prevents the use of followme because it would rule out a segment of users. How do you reconcile that? Granted, lite users, being willing to sacrifice the mngp would be alright. The second reason was that i wasnt confident i could get the full register set every second at the classics serial baud rate. Of course the full register set is not necessiraly needed, but i like to have it case bob needs debugging info or whatever. Also a comment that ross left in the newmodbus source code about it being quicker to get everything at once, rather than making 20 or 30 seperate single register calls. OK modbus is limited to 123 addresses per packet, but 4 or 5 calls still gets everything. Over ethernet you can get the whole range in a few milliseconds. Newmodbus is fast, really fast.

I suppose I'm lucky in not having a reboot for over 10 months now but it probably because I do not use the Classic's ethernet interface for any data transfers, neither local app nor Mymidnite.

Exactly.

Data to/from serial port seems rock steady and dependable.
Maybe getting a serial modbus interface from Cubie to Classic will be the way to go, I noticed that libmodbus is included in most Linux distributions and it supports serial and ethernet modbus

Its quite doable im sure, cubie has several serial ports.  For unrelated reasons my own plan is to have an atmel based daughter board. The purpose is primarily to act as a radio link / sensor hub for other things ive got going on around here. So whethers its rs232 to arduino to cubie, or direct.. Either way.

Lets see what the next firmware release brings, bob seems excited about it.

[dgd]

Whats the limit there, serial speed?

I have assumed so but thats not really true. I can see the tx/rx leds flash on the serial card almost exactly 5 times a second and that seems to fit with my modbus code reading 6 blocks of registers in about 1.2 seconds.
One limitation is the serial port ring buffer of 64 bytes which means 29 registers maximum per read hence the need for six blocks of registers.
But now, with a little research, it appears that the there is a serial_ring_buffer.h file in the arduino core with buffer size set at 64. In the Arduino forum someone has set this 256 and it seems to work. With plenty of ram on the DUE
then giving 1K of memory to the 4 port's ring buffers looks ok, no so good for the Mega with 8k ram.
some experimenting needed but if I get just 2 or 3 reads to get all the registers I need then <=1 second throughput looks possible.
I have not looked but there is probably a Linux serial port config file doing the same

Subtracting one from the other sometimes yeilds negative loads but this is only during fast changing solar conditions. There is also an outstanding bug where classic amps is wrong by several amps at  certain times. Both those things aside, load watts is still fantastic to know.

Since installing WBjr I have liked seeing load current usage when there is no PV power input. I have got into habit of turning off more wall switches.

Its erratic, it varies from once a fortnite to twice a day. The average is about 3 days. As you rightly say, i am 100% convinced that this is ethernet related. This is because the reboots only started when i started using a single open connection, and mirrors what happens when people intensively use local app which also holds a single connection.

Cripes! I didn't realize it was that serious. I though it was occasional random
I hope the final fix is just around the corner as this bug is years old.

Ethernet v serial. When i evaluated this earlier on, my decision to persist with ethernet was based on the effect on followme. It seemed to be a bad idea to design a monitoring system that prevents the use of followme because it would rule out a segment of users. How do you reconcile that?

Followme is still somewhat primitive as all it does, AFAIK, is align charge states.
With several serial to blackbox connected Classics it would be simple to read all charge states and force them all to the same state.
Also followme as now implemented is not useable when up to 4 Classics are serial linked together to one blackbox rs232 port and accessed using IDs 10 to 13 to read their registers (as also done by that MNSI interface box)

dgd

[Resthome]

I'm wondering if there have been hardware changes in the Classic that may have reduced or solved the reboot issues. I had them with an older unit that has been replaced and the while it has had disconnects I have not had reboots with it.

Zoneblue what Serial Number Classic is rebooting?

The only reboot bug I am aware with the newer Classic is if the MNGP is left on one of the Data Logging  screens and Offline Data is requested from the Local App.  That will cause a immediate reboot of the Classic.

[zoneblue]

classic 4030 ish. The reboots started around about firmware 1603. Never used to do it prior to that. Remember im making 1s modbus data requests every second of every day. Some sort of resource issue is my guess. But youve made me think about maybe closing the connection, maybe a few times a day, might help clean up maybe.

[Resthome]

ZoneBlue,

That number seem like one of the early units. Not sure what my first unit was in 2011. My currently unit is 153XX and it was a refurb replacement unit back in April of 2015. I had recurring reboots with the older unit but then discovered the Data Logging screen bug. boB also made a change about reboot that had to do with access the EPROM concurrently with some function that I don't recall at the moment but it's in one of his posts here. Wonder if that is what you are running up against.

Looks like there was a Network Compilation Changed 1597 with the 1603 firmware on 9/30/2013. Not sure what that was about or if that was the period of time people start to complain about reboot. Might be worth searching the forum to see if there are any connections.

I'm still curious as to what hardware changes may have been made since 2011.

[dgd]

Im about over the classic's SOC, its just about always wrong, given that my classic reboots so often. Time to roll a blackbox SOC data series i guess.

Just had 7 days with very variable weather and not enough power to get battery bank into float. It did get to Absorb
a few times but never completed it. SOC went between 82% and 94% although I thought this was a bit off - too low.
Then today clear skies for middle of Winter, lots of photons lighting up the pv array.
A segment of the 10 minute log records shows the transition from Absorb to Float and the SOC jumps from 96% to 100% and Ah Remaining from 990 to 1025. The Absorb charge current at 25.9 amps was 2.5% battery capacity, pretty near the ideal 3% for EA.
So overall I think boB has the SOC calculation pretty good, can't complain with just 4% SOC diversion over 7+ days.

dgd

[paul alting]

Just a short message to say I'm enjoying the detail of the posts here.

A lot of what is being mentioned pricks up my interest with my own developments in software.
With my past experiences using Arduino and also together with Modbus, I have concluded using Arduino is fine for data pass through or as low level discrete I/O interfacing.
I think like you are seeing David, using SD with Ethernet on Arduino is usually painfully slow and while the concepts are there to do it, in practice, many challenges pop up along the way.

Using a small board like Cubieboard II or better immediately allows for greater things in terms of logging.
I have Debian installed in the NAND and did have a small 40GB laptop drive connected to the SATA at one point, with that being my root file system I recall.
It worked quite well, but went back to simple NAND with the idea of using either SD or USB memory for storing logged data.

Now, Quadlog works well in NAND and the plan is to have configurable options that allow the user to specify the drive mount for all logged data to avoid the constant log writing to NAND.
It's all being stored in SQLite database.

I have often wanted to know what sort of rates do people wish to store their data ?
In my industrial projects, it has been mostly from 10 seconds up to around 5 or 10 minutes, but I don't recall 1 second rates being of any use, not unless it is a scientific application. And then it would only need short burst data logging. This is something I am thinking about. To be able to configure specific data tags to be logged in burst mode. This means, log every sample obtained from the device at what every the poll rate for that device is.

For my own logging, I set it to every 60 seconds at most. any more and it becomes too much data to really be of any use.
Another thing that is possible, and something I do, even in the Arduino, is to use CoS or CoV, Change of State or Change of Value.
I do this as part of my analog processing class and can set specific setpoints for the amount of change required to trigger a CoV flag.
If you then extend this idea out to logging, then you can see that for data values that don't really change a lot, you are not wasting space on logging.
This idea also has implications in the communications as well. If the flag is not set, then nothing needs to be communicated from source to destination, as in Arduino to Cubieboard, for example.

For Modbus RTU, using USB I have been able to set poll rates down to less than 100mSec with no errors, and a little quicker again for Modbus TCP. I think 50mSec was about as fast as I could drive it I recall.
This is using libmodbus library that I use in Quadlog.
This means that it would appear possible to have very fast logging, such as short period burst mode.
This sort of logging could be useful for issue tracking, but it needs careful attention to everything, beginning from point of measurement.

In regards to HighCharts, I have played with many data sizes and don't recall experiencing any issues with performance or display sizing, but I have found there are optimum sample sizes that appear better on the screen. Then it is a simple matter of scrolling either a full screen or half screen in either direction by making an AJAX request for that data set.
Currently I get the HIghChart data set as CSV in a HTTP AJAX request and am looking into other possibilities.
On smartphones, displaying charts can be a challenge and that will always be the case when working with physically small screens and plotting data.

All interesting ideas, I better pull my finger out and get to do more programming
____
Paul

[dgd]

I have often wanted to know what sort of rates do people wish to store their data ?
In my industrial projects, it has been mostly from 10 seconds up to around 5 or 10 minutes, but I don't recall 1 second rates being of any use

In this application, reporting on Classic throughput, I can't think of any use for anything below 10 minutes. The only possible exception being when debugging the controller hardware or firmware.

For my own logging, I set it to every 60 seconds at most. any more and it becomes too much data to really be of any use.
Another thing that is possible, and something I do, even in the Arduino, is to use CoS or CoV, Change of State or Change of Value.
I do this as part of my analog processing class and can set specific setpoints for the amount of change required to trigger a CoV flag.
If you then extend this idea out to logging, then you can see that for data values that don't really change a lot, you are not wasting space on logging.

Makes sense. However with Classic I'm leaning towards a simple stop/start loggings when daylight/dark for solar controller and no wind/is wind for wind controller. Wind will need more thought to implement.
ZB made the point that at night its nice to see power usage via WBjr but I'd do something else for that.

I'm looking forward to using the Cubie for modbus RTU or I2C connecting to an arduino that does the modbus RTU.
In any case I will be developing in C++ and will use libmodbus.  Just about all of my Arduino C code should be useable in Linux on the Cubie. I see no issues with AJAX so the  html pages index.htm (with the Canv gauges) and node14.htm (10 minutes log display)  will work as they do now with Arduino Mega server. It will be nice to have the option on Cubie of loading web pages from SD card or SSD on sata interface.
dgd

[danda]

So I am pretty happy with the one day graph.  I haven't looked at the actual data in the DB, but I've been assuming the older data is still available at the same level of granularity.

What I would like to see is 1 day, week, month, year, and all time graphs.   Each time period could also have an associated "almanac" with high/low voltage, amperage, wattage, etc, etc.

I also run weewx for logging / reporting a weather station.   It uses this type of system for breaking up into time periods, and it works well.  See screenshot at bottom. 

I'm not certain, but I believe the weewx system actually compresses/summarizes the data for each stage.  Eg, the weekly graph does not need the granularity of data that the daily does.  monthly, yearly, and all time are each progressively lower granularity.    So that's a way to save disk space, at the cost of not being able to recreate daily graphs for any date in the past.

Anyway, it might be worth looking more at how that project handles data logging/reporting for further inspiration.