Exporting data in real-time to my own website?

[TeslasPigeons]

Hello -

I am curious if there is a way to export the data from my classic 150 directly to my own website in real-time? 

[dgd]

AFAIK - No, don't think so, not over an ethernet connection, unless its been added recently.
With a serial rs232 connection there is a continuous dump of some registers in real time, you would need to write a program for your pc to read, decode and buffer this data stream.

dgd

[paul alting]

I would be interested to hear more of what you wish to do.
When you say 'export', I am guessing you mean some inbuilt function of the Classic to do the export?

Normally you have a device, say, computer connected to devices like solar charge controllers that requests the required data.
Then this computer can do what you want with the data, either make that available directly to HTTP web clients such as your web browser, by itself being a web server.
Or you can have the computer pass on the data to another computer, such as a hosted web server, to which you connect to with your web browser.
Both are available and valid.

Really depends what your needs and or wants are and what you mean by 'real-time'.
The first choice above, where the connected computer is a small web server, then you can get as close to real time as possible.
With the second option, you will introduce delays and it becomes next to real time.

As David says, the Classic can not do this on its own, you need a small computer system connected to the Classic.
That small computer system can be a very low power ARM based system that really takes very little power to operate 24/7.
You will of course need a online and continuous internet connection if you wish to be able to view the data remotely.
____
Paul

[TeslasPigeons]

I would be interested to hear more of what you wish to do.
When you say 'export', I am guessing you mean some inbuilt function of the Classic to do the export?

Normally you have a device, say, computer connected to devices like solar charge controllers that requests the required data.
Then this computer can do what you want with the data, either make that available directly to HTTP web clients such as your web browser, by itself being a web server.
Or you can have the computer pass on the data to another computer, such as a hosted web server, to which you connect to with your web browser.
Both are available and valid.

Really depends what your needs and or wants are and what you mean by 'real-time'.
The first choice above, where the connected computer is a small web server, then you can get as close to real time as possible.
With the second option, you will introduce delays and it becomes next to real time.

As David says, the Classic can not do this on its own, you need a small computer system connected to the Classic.
That small computer system can be a very low power ARM based system that really takes very little power to operate 24/7.
You will of course need a online and continuous internet connection if you wish to be able to view the data remotely.
____
Paul

Well I currently have a Raspberry Pi on the same hub as the classic 150 and it's running a shell script to grab snapshots from an IP camera I have in the area then ftps the images to a server to view remotely (on a different network, at a different location).  What I would like to be able to do is have the Raspberry Pi also read the data that goes into the exportable CSV file on mymidnite.com so I can graph and display it on my website.  I imagine there must be some way to do it if the local app can connect to the classic 150 and grab that data. 

As far as real time goes, I think it posts every 10 minutes or so?  That would be fine for a start.

[paul alting]

I imagine there must be some way to do it if the local app can connect to the classic 150 and grab that data.

Imagining these things are these days more able to brought into reality.

My understanding there are possibly a few options you might look at to get what you want.
There are a few chaps on here that are very knowledgeable, like dgd (David) and Zoneblue and others.
There are a number of projects that are being developed by these chaps that let you get connected to your Classic(s) and get the data for presentation in various forms.
Some use small boards like what you have, RPi or Cubieboard, all based on GNU/Linux, while other projects use Arduino style setups.
Both approaches offer various levels of complexity and features.

I also am developing a system that operates on small GNU/Linux boards like what you have.
With just over 12 months of development so far, there's no stopping now

What I am doing is more general purpose and fully stand alone, it is not specific to just Midnight Solar charge controllers, but will eventually talk to other components such as inverters, with work being done now on the Magnum Energy range. It is it's own HTTP web server and talks to any device using the Modbus protocol to make data available for web applications running in your web browser.
This project does not interface or have anything to do with the online 'mymidnight.com', but rather, you become your own 'yourmidnight.com' sort of site, so you have control and access locally whether you have internet connection or not.

You might like to check out the section on Open Source on the forum where some of these are discussed.
From there you might see a solution of it can be discussed more.

There's a definite growing trend of people wanting or wishing to be able to view their renewable energy system as a whole through their web browser from anywhere.
And this the sort of thing a few of us are developing and heading toward.
____
Paul

[TeslasPigeons]

My understanding there are possibly a few options you might look at to get what you want.
There are a few chaps on here that are very knowledgeable, like dgd (David) and Zoneblue and others.
There are a number of projects that are being developed by these chaps that let you get connected to your Classic(s) and get the data for presentation in various forms.
Some use small boards like what you have, RPi or Cubieboard, all based on GNU/Linux, while other projects use Arduino style setups.
Both approaches offer various levels of complexity and features.

This morning I read through quite a bit in the open source forum and a lot of it looks really interesting.   The serial connection for complete access to the registers and then the python script to decode and read is probably the best option for long-term. 

What really interested me was what RossW put together, the newmodbus, that runs in linux.  I couldn't find a really straight-forward guide, other than the tutorial on wiki which involves setting up a LAMP stack.  I'd rather have my Pi not acting as the server as the goal is to push the data from the Pi onto my web server.  Is there any where else you can point me that might have more specific instructions for the RPi?  I'll keep looking as well.  Thanks much for the help so far, it's really cool there is a way.

[RossW]

What really interested me was what RossW put together, the newmodbus, that runs in linux.  I couldn't find a really straight-forward guide, other than the tutorial on wiki which involves setting up a LAMP stack.  I'd rather have my Pi not acting as the server as the goal is to push the data from the Pi onto my web server.  Is there any where else you can point me that might have more specific instructions for the RPi?  I'll keep looking as well.  Thanks much for the help so far, it's really cool there is a way.

PM me if you like, I can probably point you in the right direction.

[zoneblue]

When you say real time, youll need to consider a few things.

The classic ethernet port is quite fast, can you can theoretically get data at around 50ms intervals, in practice there are lots of otehr considerations. I guess youve read the BB wiki, so ill just summarise:

- newmodbus is faster than any other tool, uses an open, read and close methodology, and is best suited to relatively infrequent calls, like Ross does every 5min, but the 1min cron interval limits you there as well.  I and others have tried to use it at more frequent intervals and run into problems with the classics network stack. It gets worse the quicker you go, and the slower the computer. These issues apparently are being sorted in the next firmware.
- newmodbusd which i adapted from ross's newmodbus is a daemon version, and works the same way as the local app, maintaining a single open connection, and is more suited to sample intervals around 1-15 seconds. It just maintains a file copy of the registers, which you can read and do what ever you want with. However, theres issues with this too when run at 1s intervals, something else in the classic gets upset, and more frequent classic reboots are the result.
- serial using the rj11 jacks inside the classic. dgd has done a lot of work with this and reports that none of the instablity issues noted above apply to it. However it is a lot slower than ethernet, and the baud rate makes it difficult to get a large range of registers at 1s intervals. A few maybe ok. And it ties up the follow me port.

HTH

[RossW]

- newmodbus is faster than any other tool, uses an open, read and close methodology

Just a clarification on that: newmodbus has had "continuous" mode for 2 years (since version 1.0.12):

8-Jun-13 1.0.12        Added -c for continuous operation, displays data every second,
                        writes data (if requested) on the 5-minute.
                        Added -r for "close-and-reopen" classic mode. (more co-operative
                        with other modbus applications)
and

13-Jun-13 1.0.16        Add -l (local log) and -ixxx (interval) switches

[dgd]

- serial using the rj11 jacks inside the classic. dgd has done a lot of work with this and reports that none of the instablity issues noted above apply to it. However it is a lot slower than ethernet, and the baud rate makes it difficult to get a large range of registers at 1s intervals. A few maybe ok. And it ties up the follow me port.

well slower is relative..
By adjusting the serial ring buffer size from 64 to 256 bytes I can now read over 100 modbus registers at once and a couple of reads gets all the registers I need.
This means that there is now a sub one second repeated register reading and the gauges web page I use refreshes using AJAX in under one second.
The follow-me use of the serial port doesn't matter if only one Classic is used. (theres nothing to follow!)

dgd (not intending to be pedantic)