Blackbox design

[dgd]

I managed to borrow a BBB for a week and that allowed me to get familiar with its capabilities. Returned now so just waiting on one from Element14.
I have also been thinking about a black box design using the BBB.
Basically what I am looking at is a web server, reasonably graphical not too dissimilar from the MyMidnite data screen and using the BBB for this. Web server software and lots of info on designing it for BBB is freely available.

However it is the data acquisition part that is different. I have noted the obvious frustration with the poor ethernet connection reliability of the Classic. Its being improved and boB sounds confident the 2+year quest to nail down the unreliability is coming to an end.

I have now managed to implement  modbus RTU, serial modbus via rs232, on various Arduino systems. One of these is using a minimal Arduino NANO to extract Classic data and store it is timestamped order on a micro SD card, 8Gb. This seems to work well and I have set it to take data every minute. One days data at this time resolution is stored then older data is reworked to 10 minute intervals for longer term storage. 
On viewing the data stored it may be possible to improve storage efficiency by not storing complete data if it has not changed since last reading, instead just storing a flag denoting same as previous reading. This would probably save a lot at inactive middle of night times.

The BBB supports I2C communications as does the Arduino. This is how they would communicate data, the BBB would only request stored data as needed when a client web request occurs. I can see this would allow several NANOs to talk to the BBB, one dedicated NANO for each Classic for data acquistion leaving the BBB to do the web management and serving of data.

The Blackbox would consist of the BBB and one or more Arduino NANO each name connected via rs232 to a CLassic and the ethernet on BBB being the network interface.

So far I have the NANO part all working and I have tested the i2C comms to the BBB and it works too.

Any suggestions or thoughts on this please reply 

dgd

[ClassicCrazy]

I wish I knew enough to help on this project but I have below beginner linux skills.  But as a public service I will decipher your BBB  for anyone who does not know what that means - Beagle Bone Black
http://www.adafruit.com/products/1996

[zoneblue]

I think youll find an ardunio will struggle resource wise, ram, eprom etc to do what you want it to do. Ardunino will run ethernet/wifi, just, at a pinch, but forget trying to do much with it. But there soulcd be merit in running an arduino in combination with a linux board, if you blow the nano up its only a couple bucks!

Moving up to BBB, or the new rPi2, or one of cubies, will make things easier, however best read my various scratchings on teh subject to understand the resource issues there. SD card life is a key one.  The current codeline reads each second, and creates 1 minute series from that as well as daily summaries. Still lots to do though.

[dgd]

Interesting.

I have reborrowed the BBB and having played with it plus reading a fraction of the owner comments it seems that its main redeeming factor is that it costs $45. A great low cost learning tool but I struggle to see why it is preferable to an i5 or i7 computer with several gigs of memory and a proper reliable storgae media (eg SSD).
When I look at the Arduino design it certainly seems fully adequate at being a simple web server for a Classic. After all from a 'useful' data viewpoint are we not just talking about a couple of voltage and current readings, all up maybe a dozen useful data.
Without  a proper OS it would be limited as a more sophisticated web server, but as a data collection and storage device it is an excellent choice I am working on the micro SD card storage part, nearly finished, and also an input web page to setup the various parameters needed by a lite, that is, replacing the MNLP.

I liked the I2C interface to the BBB, but maybe it is more suited to post processing Classic data into those nice graphs you have made rather than doing the data aquisition (once a second did you say?)

Then again if boB fixes the Classic ethernet reliability then all my efforts with modbus RTU will be  somewhat redundant.

dgd

[mtdoc]

I think the advantage of using the BBB or similar lies in the much lower power consumption. Obviously one faces a choice, depending on needs, requiring a balance between processing power, memory/storage and power consumption. Of course price is a factor as well.

For me the utility of an arduino or similar low cost/low power MCU is as an intermediary data collection/serial interface between my Classics and a more powerfull computer where I can integrate and display data from both Midnites equipment and my Outback equipment.

Which more powerful computer I use - BBB, rPi, other small ARM based solution, Atom based netbook, or full laptop depends on how i choose to balance those factors.

[TomW]

Was hoping to participate in the BBB end of this effort.

Problem is I just sent back my second failed Element14 BBB. Both failed with same boot error about accessing pin 21 on device. Seller or internet could offer no info on what it is hardware or software, me or the BBB  but I am not going for a third unit.

Still got a couple rPi's tho.

Too bad, the BBB were nice machines  while they worked here.

Not sure how long before I get any permits and stuff for a system at the new place but Spring at the earliest.  That means nothing to monitor until then. 

Will be watching the progress, however!

Appreciate you guys sharing this stuff.

Tom

[zoneblue]

Well the cubieboard i originally bought to try out has been running 24/7 for going on two years now. I rebooted it maybe twice, and replaced the (cheaper) sd card once (with a better one). There are now 4 differenct cubieboards to choose from , see ebay, amazon, or various places. The only thing is They come with android by default, so you need to download and burn a "Cubian" debian OS image, but thats pretty easy, xplained here: http://zoneblue.org/cms/page.php?view=cubieboard-and-debian

[dgd]

I have followed ZB's development of the black box concept and the use of a BBB in that project. The graphs he produces from collected data look very professional. The usefulness of them is not doubted as more than a few times they have been used to identify anomilies and possible or real problems with the Classic.
To collect sufficient and ongoing data that means the BBB is probably permanently running and connected to a Classic via tcp/ip, of course using up the only connection the Classic can provide. 
I also understand the desire for lowest power used by any permanently connected device.

I will hopefully soon own a new BBB. The get the Arduino data collectors I2C connected to it. An important aspect of this project, for me
is that the only permanently connected devices will be an extremely low power Arduino with micro SD, serial and ethernet on EACH Classic.
The BBB will only collect data from these when I want to post process it into graphs etc.
I'm not sure I would want to have a 24/7 BBB online as a web server as I dont see the need (for me) to web server out analysis graphs etc.
The simple snapshot, refreshed every 10 seconds, with the Arduino web server would be sufficient. Maybe a second page of a the days summary of data.

dgd

[zoneblue]

To collect sufficient and ongoing data that means the BBB is probably permanently running and connected to a Classic via tcp/ip, of course using up the only connection the Classic can provide. 

Well teh point of BB is to provide data to other users, hence you dont need another connection. Mymidnite is always free, so thats an option.  You also dont have to tie up the port if you do read, open close. And, well theres nothing to stop you connecting over serial, those arm boards have several UARTs on baord, plus a dedicated TTL port, plus usb. Take your pick. However note than rs232 is slower, and there wont be much if any time left over once you have read the full range of register range each second, to do much with it. If you can be more picky about which registers you want and just buffer the data, for processing later, itll work fine. Seems sensible if you are targeting lite uses with no mngp to compete with.

Another option is of course if the ethernet stack issues get resolved is to use  a connection that only opens breifly, allowing others in. Ive built a function into BB, that disables it, long enough to use the local app. However, i think a better route is to build config into BB, and avoid teh need for local app.

I also understand the desire for lowest power used by any permanently connected device.
I will hopefully soon own a new BBB. The get the Arduino data collectors I2C connected to it. An important aspect of this project, for me
is that the only permanently connected devices will be an extremely low power Arduino with micro SD, serial and ethernet on EACH Classic.

Id be interested in real world power figures for BBB. As for arduino, low on its own but did you measure it with ethernet sheild? Most/all ethernt ports use about half a watt for the isolation circuitry. Id also be interested in figures from your sheild. Personally i dont think 1.25W is that onerous for the BB, not when it gets you access from as many clients as you want anytime, realtime charting etc.

The BBB will only collect data from these when I want to post process it into graphs etc.
I'm not sure I would want to have a 24/7 BBB online as a web server as I dont see the need (for me) to web server out analysis graphs etc.

Wells there is a precendent for your proposed architecture. Typically these sort of solutions would use a hosted server in a datacenter to do the UI side of it. Gets you simple access from anywhere in the world, and the microcontroller just syncs hourly on a push basis.

Linux baords take ages to boot, and better left on. Thats their thing. Id youre going the home server route, youd might consider not using a dev board but something like a atom NUC, something more capable, and that can be used for other things.  Well hey, you can use it for weather station, home security/automation, skys the limit.

[zoneblue]

On the dev board front there is a brand new rPi out, which brings it more into line with the sort of specs that BBB/cubie have offered for over 2 years. Still no sata though. Theres a Pi knock off called banana pi that has sata, and gigE. 
http://linuxgizmos.com/sbc-mimics-raspberry-pi-has-faster-cpu-adds-sata/

I had a quick look for cubieboard2 on ebay, USD 65 bucks delivered.
http://www.ebay.com/itm/Cubietech-Cubieboard-Allwinner-A20-SOC-MiniPC-Cubieboard2-extended-pin-interface-/321406619711?pt=LH_DefaultDomain_0&hash=item4ad553d03f

and aliexpress have a good deal going:
http://www.aliexpress.com/item/New-Arrival-PC-Cubieboard-A20-Dual-core-Development-Board-with-Power-Cable-SATA-Wire-USB-to/1634976465.html

USD 63 and change gets you the usual stuff plus a free case, and a free USB-serial adapter (those are useful extras worth about 20 bucks)

For those that noted my x86 preference, what became of atom silvermount, you ask... well, intel went its pricey consumer ddriven settop NUC route, which is a pity becasue they used to make some nice mini itx atom stuff.

However a couple of products caught my eye:

MinnowBoard Max
http://liliputing.com/2014/03/minnowboard-max-99-intel-atom-powered-single-board-computer.html
http://www.minnowboard.org/meet-minnowboard-max/

Seems to be a community driven intel cpu dev baord. Interesting. E38xx series CPU, 22nm bay trail embedded. thermal around  6W nominal. Looks interesting.

Intel did briefly make a bay trail NUC, but later upgraded it to haswell.
http://www.legitreviews.com/intel-nuc-dn2820fykh-bay-trail-system-review_135053

Asus Vivomini
http://www.asus.com/ASUS_VivoPC/VivoMini_UN62/

Another NUC, but on teh cheaper end. Haswell chip too.

If ARM agravates you then these sort of intel based products are options for little extra cash.

[mtdoc]

On the dev board front there is a brand new rPi out, which brings it more into line with the sort of specs that BBB/cubie have offered for over 2 years. Still no sata though. Theres a Pi knock off called banana pi that has sata, and gigE.  http://linuxgizmos.com/sbc-mimics-raspberry-pi-has-faster-cpu-adds-sata/

Don't forget the Odroid products

Their C1 is only $35 and it has better specs that the rPi2 (though not the extensive user community.)

There's lots of options out there.

[dgd]

The spec for that cubic 2 board was just nice for what I want, ESATA and I2C as well as decent dual core CPU and a gb of ddr3.
Bought one and won't bother with the BBB now 

Dgd

[dgd]

I looked through the info on the progress of the blackbox project and noticed a quite high sampling rate, using newmodbus, of the Classic registers. It appears that a one second rate is being used.  Why so many samples? would one per 5 seconds or even 10 seconds still not provide sufficient detail AND catch any important events?

dgd

[zoneblue]

Good question. It is adjustable. What happened was this. When i was doing 60s samples using ross's original newmodbus, (open read, close) the classics network issues would drop about 5% of the samples, and then lockup after about 14 hours. With 3-5 minute samples i could laregly make that all go away. However at the time i was working with cron which has a one minute logical sample rate. In rewriting newmodbus to work as a daemon (open read and stay open), that gave me sub 60s ability. I asked andrew and he said 50ms was the lower limit, and so i just thought well 1s seems the obvious candidate. That would give us two things: a real time data, using ajax type UI functionality. When you turn a heavy load on , youll see the results more or less instantly. Not have to wait a minute. Secondly, the classic firmware, has some trouble tracking soc, due to the reset issue. I figured i was going to have to write my own SOC datapoint, and that would require fairly quick sample rate to get the accuracy.

Thats the short answer. 1s sample rate does increase power use, network traffic,storage requirements and other things. Local app uses 2s sample rates. Thats a possibility.

Currently what happens is the 1s data is put into a ram drive where it sits for 60s, when the last result is inserted into the database. My intention was to add averaging for that at some point. Then once a day the entire days 1s data is agressively compressed using bzip and written to sd card. (this is the entire register range which is useful for debugging purposes, or study of unusal load events.) I plan to further reduce the database writes by storing the 1s samples to flatfiles. They are static, with no need to be rewritten, hence much of the advantages of db storage are lost. Instead the db will be used for summarys and agregations.

[xsnrg]

@zoneblue, have you considered using rrd for your data storage?  It's back-end data store has been very compressed and optimized over its long life, and it can inherently take care of the tasks of averaging datapoints, configured to do so based on true rolling average, average peak, etc.  You can also extract the data back out of it quickly if you ever need to.

[url]http://oss.oetiker.ch/rrdtool/[url]

Nice work on the project.

Jim