Blackbox design

[dgd]

I was looking at 5 readings per minute (1 per 12 seconds), averaging to one minute, storing  and I2C sending  from Arduino Nano to Cubie.
Ok, it might miss some fast changing events but for reporting/graphing purposes that would not be important

The ATmel 328 has 1K of eeprom and from the specs I see it has 100,000 writes life (the 32k flash for program storage, has 10,000 writes)
So working with a 32byte maximum, but probably smaller, record being stored, that eeprom space would accomodate 32 records, ie 32 minutes of data before starting to overwrite.
One write per 32 minutes would make a life of 100,000 * 32 minutes, 3,200,000/1440 days which is 6 years
Reducing the record size or increasing the one minute dt resolution would significantly increase that six years although replacing the NANO would be no great deal 

This would avoid that whole bothersome SD card thing and its potential problems.
I could envisage the data from the NANO being taken by the Cubie and stored on SSD media on its esata interface. That would take a load off the ethernet

dgd

[zoneblue]

rrdtool is primarly a graphing tool. Thats all handled by the BB web app, and likely to go javascript graphing ultiamtely.

I considered a lot of things, in memory databases included. Much of the decision boils down to how many writes there are to sd. If you can get that down to a megabyte every 3 hours, the sd card should last. Much else and it wont.

@zoneblue, have you considered using rrd for your data storage?  It's back-end data store has been very compressed and optimized over its long life, and it can inherently take care of the tasks of averaging datapoints, configured to do so based on true rolling average, average peak, etc.  You can also extract the data back out of it quickly if you ever need to.

[url]http://oss.oetiker.ch/rrdtool/[url]

Nice work on the project.

Jim

[RossW]

rrdtool is primarly a graphing tool.

Disagree, entirely!

The RRD in RRDtool stands for "Round-Robin-Database"
rrdtool is first and foremost a set of tools for the collection, summary and storage of data.
The entire rrdtool PACKAGE includes rrdgraph, which is a graphing engine that primarily uses data from various rrd databases, along with calculations if required, to produce graphs.

(I've been using it extensively for almost 15 years)

[zoneblue]

Each to their own, ross!

Not sure how any of that helps, dealing with write sensitive media, and  needing to do a lot of processing and presentation, ie web app, that rddtool cant do? Go ahead, enlighten me!

[RossW]

Not sure how any of that helps, dealing with write sensitive media

I'm not saying it actually helps, I'm just saying that "RRDtool is a primarily a graphing tool" is manifestly wrong.

RRDtool (rrdgraph) can generate graphs ON THE FLY without any writes, so in that regard, it's not unhelpful.
I do agree however that anything that needs to be able to store data in a nonvolatile manner, with limited disk write life, is a challenge. The best I can suggest is some I2C or SPI battery-backed-RAM. With unlimited write cycles and very fast access times, it's the answer. With a supercap, it should survive weeks without power, maybe years. Yes, it's more bits and more expense, but just possibly the small additional initial cost is outweighed by the significant operational benefits?

[ClassicCrazy]

I like the discussion because I never would have had a clue to all the design and programming that goes into makes hardware and software function. 
 
This and the other networking discussion going on is fascinating to see as an outsider to understand all the technical details involved.

[dgd]

It seems I have been caught by the Chinese new year holidays as my Cubie has estimated delivery at end March to early April. Bummer 
So to keep things Blackbox interesting I bought an Arduino Mega2560 to play with, gave up then got an Arduino DUE. This is much more interesting as its based on the Atmel ARM Cortex M3 cpu.

I was trying to get one web page that displayed info and running data on two or more Classics but even with the Mega2560 4 h/w serial ports and an rs232 connection to each Classic the task proved near impossible. Running two or more parallel modbus processes (state machines) was just not going to work because the modbus RTU timing and retrying is very timing critical. Definitely in the bloody difficult category 

Anyway this is not really what modbus libs were designed for. The topology is for multiple slave devices on one bus where each device ID is unique and a master's state machine can poll them as required.
The Classic only provides rs232 serial which is a point to point physical layer so proper multi system polling over rs232 appears (is) impossible.
So to progress with this use of serial communications I have installed several rs485 converters on Classics and have an rs485 arduino shield that I used on the Mega2560
RS485 is a serial connection which can connect multiple other RS485 devices to a shared physical cable (bus) - better descriptions on google

The RS485 for Classic is a bit more tedious to implement. I have used a small MAX232 pcb (that also has the needed max232 circuitry) to shift rs232 signals to TTL logic levels then connected another TTL to RS485 converter ( that uses a MAX485 and required circuitry).
These tiny converter circuit boards are from Ebay and cost a dollar or so each - and most importantly they do seem to work.

The next stage is now getting the modbus lib on the Arduino M3 to extract the data from several rs485 bussed Classics and present it in a not so simple web page.
I have noticed that even just the 16mhz Mega2560 cpu can do a simple web server page refresh rate of about once per 4 seconds, Getting pretty close to a real time display.
The first couple of tests using rs232 on the 84Mhz Cortex M3 get this to nearer every 2 to 2.5 seconds, I have not done the sums yet but maybe this is approaching the rs232 baud rate capabililty or past it already, or maybe not 
Perhaps if I replace the sketch usage of FLOAT types and do string/integer manipulation to display Classic data numbers not only would the annoying trailing real number zero after the decimal point be gone but the script might run even faster for a few millis faster page refresh rate 

dgd

Cheeky note for boB, next revision of Classic change one of those rs232 serial ports for RS485, betcha its dead easy to do 

[ClassicCrazy]

A lot of this is over my head DGD but it is something I want to understand.
You connect the Arduino DUE to Classic serial  - same one that the firmware programming  uses ?

Is the reason you are using the RS485 because you want to talk with more than one Classic ?  Otherwise just RS232 port would be good enough for one Classic ?

The Arduino then runs the modbus program and asks for data , such as voltage register  and gets this every 2 seconds . When it gets it then the Arduino program will convert it to what we can read as voltage . Then what happens to the data ?  Displays on monitor attached to Arduino and or another program running on a Linux computer like Raspberry Pi or Beaglebone Black will make this available on web server as a graph  ?

Thanks,
Larry

[dgd]

Larry,
The arduino connects to one of the rj12 sockets which you can see when the cover is removed from Classic. Usually the MNGP or MNLP plugs into top one and the other two get used for follow-me.

Rs485 because it's a bus type wiring so multiple classics could connect to blackbox.
It seems to be an industry standard with modbus communicating devices. The classic, however, went with simple 3wire rs232, I assume because it was only intended to connect an MNGP and any multi classic setup would use ethernet for data gathering.
(Being an ever optimist I was waiting for boB to reply saying that rs485 was already available by writing some undocumented configuration register(s) then one rj12 would become the 4 wire rs485 connector)

The arduino becomes a modbus master device and requests data from Classics slave.

The data can then be stored in some way, short term eeprom, SD card, USB drive etc.. Then possibly sent to a more suitable computer system for longer term storage, analysis and reporting/graphing.
Eg the excellent graphing/reporting from zoneblue

No monitor attached to arduino. It can present the data, almost in real time, via a web page. You just connect your web browser to the arduino and a snapshot page of data, refreshed every 5 seconds, appears. It's not a very sophisticated display, no graphics with analogue meters etc. but contains all the essential data to monitor a working Classic and battery bank if WBjr attached to classic.
For a single classic the arduino web server works nice, as detailed in that forum subject.

The web server that integrates data from several classics in a similar page is where I'm more interested. Almost done too so will hopefully get full description of hardware and program listed under arduino forum soon 

Dgd

[ClassicCrazy]

   Okay - thanks for explanation. Hopefully in the future there will be some kind of  do it yourself port so all the communication and monitoring hackers can have at it without too much fuss. Seems like if there was a simple jumper or hardware modification to enable the port that might take care of security concerns - only those who made the mod would have access.