I am able to redirect the data streams that emanate from my Classic destined for the MyMidnite server.
The data appears to be encrypted and I need to decrypt it or have the encryption process in the Classic disabled.
Is there an modbus register than can be set to do this?
If not, then what information do I need to do the decryption on one of my network computers? and can a future firmware update include such a register?
Also any info on the data format of the data that streams to the MyMidnite server every 10 minutes would be useful.
dgd
Hi, DGD,
The data stream is just modbus encrypted using 3DES. The keys are unique for each Classic. There is no method for turning the encryption off at the moment as the information is already available decrypted through the standard MODBUS interface. If you are already redirecting the data stream you might be better off using your redirection to use MODBUS read your own set of registers and then send that data where ever you want it.
Thanks,
-A
Andrew,
I am already accessing raw modbus data but with the Classic limitation of one tcp connection I realised the MyMidnite was using a second tcp connection not available for other processes.
Now if I could make use of this encrypted data I could have two reporting programs accessing just one Classic - one live data dashboard display and the other a reporting grapher.
dgd
That's true. Unfortunately it would be very involved to redesign the mymidnite interface to make for an on/off encryption stream. It sounds easy but there is a whole bunch of handshaking that goes on to establish the connection. I don't think we'll be able to offer an encryption-free stream in the short-term.
Or if you had the DES key you could decrpyt it... Still for most datalogging purposes the fixed 15 minute interval is likely to be too long, yeah?
Well that's true but you'd also have to implement all the handshaking in order for the Classic to talk to your application as well. We were intending to build a PHP package that you could install on your own web server that would do all that for you but we're still swamped at the moment and that is down the bottom of the list.
It seems to me that the "combox"/blackbox route solves all of these problems.
However given that most folk need plug and play, its either gona have to be a midnite undertaking, or a very comprehensive community one.
In terms of the community effort, the various hacks and patchs that we are all doing is certainly not fast tracking development on this, and no faster than the midnite timeline.
What to do?
Quote from: atop8918 on March 18, 2014, 03:15:20 PM
Well that's true but you'd also have to implement all the handshaking in order for the Classic to talk to your application as well. We were intending to build a PHP package that you could install on your own web server that would do all that for you but we're still swamped at the moment and that is down the bottom of the list.
Handshaking processes are quite normal in data communications applications.
If some documentation were provided on the existing handshaking requirements that currently exist between the Classic and Mymidnite server along with the appropriate decryption key information then I have no doubt we (users) could do something useful.
Please take 10 minutes to provide this info.
Thanks.
dgd
Quote from: zoneblue on March 18, 2014, 03:26:36 PM
It seems to me that the "combox"/blackbox route solves all of these problems.
However given that most folk need plug and play, its either gona have to be a midnite undertaking, or a very comprehensive community one.
Sadly neither of these options are viable.
Midnites often mentioned paucity of resources means glacial progress on their non
front line undertakings.
The community efforts are laudable but with very very few with the expertise to complete a commercial grade
combox for MN products (Classic and Kid) then this approach too is sporadic at best.
Nevertheless its probably all we will ever have.
dgd
Well you know what they say. You cant please everyone.
We have tried to make a very nice charge controller that works well and has functions that everyone wants. Of course that's not possible so we do as much as possible and also keep it affordable. Its true there are things it does not do that some would really like but then there are others who like it as is. So at the end of the day we have to look at a few factors to decide what to do. We have to decide where our money is best spent if you will. One goal is obviously to make enough money to stay afloat. With that in mind a company really needs to balance Feature creep and new products. It is very easy to spend hundreds of thousands of dollars on feature creep for 1% of the users.
So with that in mind the Classic is not done by any means but we are focusing the majority of the engineering resources towards new products. We will continue to make changes and add features to the classic though and we truly value every ones input.
So if we have severely let down any of our customers I truly apologize but we are balancing our resources as we see best.
I will chat with Andrew about the handshake data and see if I can get that info.
On a side note we are going to put a call out on the KID for some one to program a "communications adapter". Our goal is to have it speak modbus like the classic and maybe in that box there is enough resources to fulfill a lot of requests. As soon as Mario gets some serious serial data flowing We will start a thread on this and see who can help move that project forward
Ryan
Thanks for your reply Ryan.
My goal here is to encourage some more open information on the technical side of the Classic.
Although the Classic's comms limitations are well known (ie one tcp connection) this does not detract from the fact that the Classic is a well featured and quality solar controller. I have not seen anyone disappointed with it yet.
So please don't think I am somehow disappointed, quite the opposite in fact.
Now that I have an opportunity to play (for 2 days) with an eTracer controller and just started with a Swiss made Studer innotec controller I can even more appreciate how far out in front of competition Midnite is.
Just the addition of the WBjr and soc info leaves everyone else way behind.
On that side note, have you (MN) drawn up a general specification for the proposed MNcommbox?
I'm assuming you need the processor to maintain memory registers compatible with the modbus spcecs or MN version thereof. Serial input/output via USBs to Classics and KID? and perhaps data archiving.
Maybe an inbuilt web server? ;)
dgd
Why isnt supporting say blackbox an option? You seem to have energy to try to decrpyt the mymidnite stream, and hack about your system generally.
The reality is that the classic really has a vast array of communications options, more than any other device. True none of them are "fisnished", and it could be said that resources are being spread thinly over too many options!
Quote from: dgd on March 18, 2014, 06:39:43 PM
Quote from: zoneblue on March 18, 2014, 03:26:36 PM
It seems to me that the "combox"/blackbox route solves all of these problems.
However given that most folk need plug and play, its either gona have to be a midnite undertaking, or a very comprehensive community one.
Sadly neither of these options are viable.
Midnites often mentioned paucity of resources means glacial progress on their non front line undertakings.
The community efforts are laudable but with very very few with the expertise to complete a commercial grade combox for MN products (Classic and Kid) then this approach too is sporadic at best.
Nevertheless its probably all we will ever have.
dgd
Quote from: Halfcrazy on March 18, 2014, 08:00:17 PM
On a side note we are going to put a call out on the KID for some one to program a "communications adapter". Our goal is to have it speak modbus like the classic and maybe in that box there is enough resources to fulfill a lot of requests. As soon as Mario gets some serious serial data flowing We will start a thread on this and see who can help move that project forward
Ryan
It might make sense to have such a thing be able to talk to both Kid and Classic?
Quote from: zoneblue on March 18, 2014, 09:01:35 PM
Why isnt supporting say blackbox an option? You seem to have energy to try to decrpyt the mymidnite stream, and hack about your system generally.
The reality is that the classic really has a vast array of communications options, more than any other device. True none of them are "fisnished", and it could be said that resources are being spread thinly over too many options!
Ok, I'm not being critical of anyone's efforts here BUT
the blackbox concept seems to centre on a physically small limited resource single board computer. Albeit one really useful for a process control application but more often touted as an educational tool.
Having bought into the rPi and its Linux, I quickly discovered its unreliability and performance issues.
I gave up and its in the junk box. Great educational tool (its primary purpose?) but doing anything real world is painful.
Maybe the others are better.
So my thoughts moved to the blackbox being a software project. Just run it on a proper computer with decent memory, storage, cpu power etc and the project reduced to just reading modbus via serial port or TCP.
No critisism of Ross but I didn't really want a command line modbus interface, I wanted to compile modbus functions into my higher level reporting progam, written in C,C++ I know I could have simply used a 'system' call within my code to Ross's newmodbus but the performance issues with exiting to a shell command and paramter passing is something I don't want.
My coding developed into my simple C modbus library for Classic. Other C libs for modbus just were way too convoluted and complicated for me.
Now to the Classic, no matter how smart the code is I write and does what I want, the overwhelming and crippling limitation of the Classic is its ONE and ONLY TCP connection. I cannot have a permanent PC connection via TCP to gather and live report or archive data as I can then not connect remotely with say the Local App to get a snapshot of how things are going.
This is why I am interested in this MyMidnite connection.
I don't use MyMidnite, its too limited for me to find useful (no negative reflection on developers intended)
BUT it does have a second TCP connection reserved for it in the Classic.
I would like this TCP connection made available for other purposes.
Ideally the Classic firmware has its TCP connection limit changed for the current setting of ONE to a setting of TWO.
Failing that then change the Classic firmware to allow the 'web access' system to be defined by the user - an MNGP function or Local App function for Lites. Or the server name setup in a series of modbus registers so that those of us who prefer that method of changing the server name can do so.
Failing that then the required information on the Encryption key and the handshaking the Classic is expecting. Its a dogs bollix of a job but it is possible to redirect the data to the MyMidnite server to a local network server.
Really the proper solution is the first option, increasing the TCP connection limit to TWO as we already know the Classic can deal with two connections.
dgd
QuoteOk, I'm not being critical of anyone's efforts here BUT
the blackbox concept seems to centre on a physically small limited resource single board computer. Albeit one really useful for a process control application but more often touted as an educational tool.
Having bought into the rPi and its Linux, I quickly discovered its unreliability and performance issues.
I gave up and its in the junk box. Great educational tool (its primary purpose?) but doing anything real world is painful.
Maybe the others are better.
Actually ive been quite impressed with Cubiebaord. But theres nothing at all stopping you running blackbox on any linux box, indeed ive also run it on my atom dev server, which draws 20W. It ran little better and drew 15x more current! What is it exactly you are trying to achieve that needs 2Ghz cpu and 8G ram? Theres some nifty new atoms coming out that may be a good comprise.
Quote
So my thoughts moved to the blackbox being a software project. Just run it on a proper computer with decent memory, storage, cpu power etc and the project reduced to just reading modbus via serial port or TCP.
No critisism of Ross but I didn't really want a command line modbus interface, I wanted to compile modbus functions into my higher level reporting progam, written in C,C++
I use newmodbus to talk to the classic via modbus, its way way faster than anything else out there, and rock solid. C is the appropriate language for that for sure. But i wouldnt code a web app like blackbox in in C, not in a million years. The work ive done on newmodbus, gets around the stack timeout issue, gives faster and more stable timing and now handles classic reboots gracefully. It transfers the entire register set to a text file, one address per line. All the web app has to do is read the file. But did i know C before i started that side project, no i didnt. Sounds like you know it though, and i couldve used your help.
QuoteI know I could have simply used a 'system' call within my code to Ross's newmodbus but the performance issues with exiting to a shell command and paramter passing is something I don't want. My coding developed into my simple C modbus library for Classic. Other C libs for modbus just were way too convoluted and complicated for me.
Well isnt that us all hacking away and not benefiting from the synergy we could acheive by cooperating? To realise a community supported combox will require our mutual best efforts employed. Im not saying my way is the way, far from it, i merely opened up some code as a discussion point.
QuoteNow to the Classic, no matter how smart the code is I write and does what I want, the overwhelming and crippling limitation of the Classic is its ONE and ONLY TCP connection. I cannot have a permanent PC connection via TCP to gather and live report or archive data as I can then not connect remotely with say the Local App to get a snapshot of how things are going.
Thats the whole darn point of the combox, to allow multiple conections! The webserver in the box allows mulitple simulatnous connections from anywhere, and talks to the classic using its one free modbus link.
QuoteThis is why I am interested in this MyMidnite connection.
I don't use MyMidnite, its too limited for me to find useful (no negative reflection on developers intended) BUT it does have a second TCP connection reserved for it in the Classic.
I would like this TCP connection made available for other purposes.
Ideally the Classic firmware has its TCP connection limit changed for the current setting of ONE to a setting of TWO. Failing that then change the Classic firmware to allow the 'web access' system to be defined by the user - an MNGP function or Local App function for Lites. Or the server name setup in a series of modbus registers so that those of us who prefer that method of changing the server name can do so. Failing that then the required information on the Encryption key and the handshaking the Classic is expecting. Its a dogs bollix of a job but it is possible to redirect the data to the MyMidnite server to a local network server.
Arent these the 1% type asks that Ryan is talking about? Id possibly go along with: if mymidnite is disabled, to allow a second modbus connection, if its practical to code. Itd make my life easier at the moment yes, but its not a deal breaker honestly. My present plan is pretty simple, to have a button on blackbox UI, that says enable/disable the classic connection. Using the local app is then a matter of clicking disable, doing your business then renabling it. Its not ideal but will work. As i said the other day, it would also be possible to build the local app config functions into the blackbox, and be free of Air for good.
Quote from: zoneblue on March 18, 2014, 11:01:07 PM
These are all the 1% type asks that Ryan is talking about. They already have several ways to talk to it and you want several more. Id go along with: if mymidnite is disabled to allow a second modbus connection, if its practical to code. Itd make my life easier for sure, but its not a deal breaker honestly. My plan is real simple, to have a button on blackbox UI, that says enable/disable classic link. Using the local app is then a simple matter of clicking disable, doing your business then renabling it. Its not ideal but will work. As i said the other day, it would also be possible to build the local app config functions into the blackbox, and be free of Air for good.
I only want that second TCP connection made public. Pity if it is that 1% ask category.
The blackbox is another separate project, possibly a longer term development and I agree the possibilities are interesting.
Maybe we should have an MN combox discussion, separating possible harware specs from software requirements. Maybe most of both is done already.
dgd
BTW the latest newmodbus (daemon version) is here:
http://code.google.com/p/theblackboxproject/source/browse/trunk/c/newmodbusd/source/newmodbusd.c
I agree some planning is a good idea.
The objectives document that ross and i threw together is here:
http://code.google.com/p/theblackboxproject/wiki/Planning
Its a wiki, if you want a login i can sort that out. Threres also a lot of discussion on the hardware side in the subforum, here, that is yet to be rationalised and included.
Hi, guys,
These are all good points and requests. Again, we're not trying to hold out on you by being stubborn: it takes a lot of time to do these things, otherwise we would have them done. For me to accurately document the handshaking process would probably take a day or two of work including getting back into the code and making sure there are no changes since the original design document. This is also Company Proprietary stuff. If I pass this out then competitors also see it and it is part of our IP meaning we could face competition right away. Also by handing out encryption keys, which though I truly believe belong to you as the customer, we run the risk of malicious folks getting their hands on them and starting to (really) hack our system (I spend a lot of time thwarting hacking on the website as it is now). In addition to all of this I will also be tasked with answering any questions relating to helping 3rd parties try to get the handshaking to work. I simply do not have any hours left in the day to do this. 16 a day is all I can manage and still have a little time for my children and some sleep once in a while.
We designed the communications in the Classic to compete with our nearest competitors and provide two plug-and-play interfaces: the Local Application and MyMidNite. If these are not useful to some customers then the weight of additional features unfortunately rests with you (unless you can pay us for the additional development time and resources or buy 1000+ units). We are open and willing to provide any features that we can fit into the schedule and will provide us with additional sales to compensate for the development time, but we can't spend a lot of additional time on features that won't improve our sales -- that makes us lose money which we really can't do at this point.
To add another TCP connection I would have to implement system-wide mutex locking. I would have to re-architect the modbus stack. In fact, I would have to do all the things that were deemed too time-consuming and unnecessary way back when we were initially designing the communications. So I'm afraid one connection is all that there will ever be on the Classic. Just like one Start button on Windows, or one App Store on iOS, or one screen on the iPhone.
That having been said, please, please, please keep suggesting features. You guys are awesome and have pushed the Classic along very well with many of the things you have thought of. We do value your opinions and suggestions, we just cannot possibly implement them all.
Quote from: dgd on March 18, 2014, 10:02:15 PM
Now to the Classic, no matter how smart the code is I write and does what I want, the overwhelming and crippling limitation of the Classic is its ONE and ONLY TCP connection. I cannot have a permanent PC connection via TCP to gather and live report or archive data as I can then not connect remotely with say the Local App to get a snapshot of how things are going.
dgd
You guys do realize there are unlimited RS232 connections available? It is independent of the TCP/IP so no matter how bad we may have "Crippled" the controller there is still another port.....
I wonder if a Beaglebone Black or a Raspberry Pi could connect to both TCP and serial at the same time and display all the data via webserver ? I am still novice learning the Linux world but that seems like it would solve some of these issues inexpensively.
Quote from: Halfcrazy on March 19, 2014, 05:40:18 AM
You guys do realize there are unlimited RS232 connections available? It is independent of the TCP/IP so no matter how bad we may have "Crippled" the controller there is still another port.....
I was put off put off by a couple of folk that said they tried using serial and found it error prone.
Where would we find documentation for the serial port? Im interested in testing it, for one thing it will save 2 watts or so.
Could we also have an updated modbus register doc, as there are now *way* more registers than the May13 doc.
I am curious - what is the two watt savings from ?
Quote from: zoneblue on March 19, 2014, 04:03:03 PM
I was put off put off by a couple of folk that said they tried using serial and found it error prone.
I had these issues but it was an rPi limitation not the Classic itself. It basically just dumps the data so if your collection machine has to use resources for something else on the communications chip (serial,ethernet) the data can get corrupted / truncated.
From memory.
Tom
Yes the RS232 should be very stable.
The 2W is basically the ethernet switch needed to link the classic, blackbox, and the LAN.
The blackbox itself only uses 1.25W, but that also includes about 0.6W for its plugged ethernet link.
Then theres whatever the classic uses maintaining its end of the connection. So total could be higher than 2W.
Glad to hear you analyze power consumption use in watts instead of amps. Every day I see so much power wasted everywhere where I work it would almost be a joke if I suggested saving a watt. You must have a very efficient system.
Quote from: Halfcrazy on March 19, 2014, 09:01:54 PM
Yes the RS232 should be very stable.
It is but slow 9k6 or 19k2 modbus throughput.
There is not a lot of real time processing you can do at this comms speed
Quote from: zoneblue on March 18, 2014, 03:09:41 PM
Or if you had the DES key you could decrpyt it... Still for most datalogging purposes the fixed 15 minute interval is likely to be too long, yeah?
The interval appears to be a function of the handshaking. In early days of Mymidnite release there was a while when a one minute interval was running. That made the data very useful.
Unfortunately my suggestions to have a user specified interval time from 1 to x minutes was rejected, even though this would mean just a couple of days data retention instead of 2 weeks worth with a 10 minute interval.
This is one of the first changes I would make if able to redirect handshaking to a local server.
dgd
Quote from: atop8918 on March 19, 2014, 04:30:08 AM
Hi, guys,
These are all good points and requests. Again, we're not trying to hold out on you by being stubborn: it takes a lot of time to do these things, otherwise we would have them done. For me to accurately document the handshaking process would probably take a day or two of work including getting back into the code and making sure there are no changes since the original design document.
I know that you know as a professional systems/applications programmer that maintaining up-to-date documentation is essential. I'm sure that if MN management were made aware of the necessity of this task that they would allocate your resources to complete this necessary task.
Then providing this info would be so simple.
Quote
This is also Company Proprietary stuff. If I pass this out then competitors also see it and it is part of our IP meaning we could face competition right away.
Ok, I have never known any form of handshaking protocol or sequence be a significant risk to a company survival. However if MN management share your view then this becomes an immovable blocker to any further progress on opening this Mymidnite communications channel.
I would greatly encourage you and MN to lean more to open documentation (and source but MN already closed this idea)
Quote
Also by handing out encryption keys, which though I truly believe belong to you as the customer, we run the risk of malicious folks getting their hands on them and starting to (really) hack our system (I spend a lot of time thwarting hacking on the website as it is now).
I cannot understand the purpose of the data encryption. Only the actual data items appear to be encrypted as I'm not aware that the TCP header/routing info is encrypted.
So the only purpose of this appears to be to deny the owners of the data from using any security or data comms analysis tools to confirm that only performace data/Classic ID info is actually being transmitted.
I think you have this the wrong way around.
If, as you say, you already spend time dealing with malicious hacking of the server then my main concern would be that the server becomes compromised, or already has been, and malware/spyware is already installed there OR the server is linked to another compromised/malware server.
If this does happend then the server has a hidden and somewhat secure data link to a network device on customer's networks.
I can't see the unencrypted data stream so the Mymidnite server could be being used to investigate possible security openings on networks the Classic is connected to.
This however is not why I want decryption, I just want access to my data on my network then I can start doing something useful with it.
Quote
In addition to all of this I will also be tasked with answering any questions relating to helping 3rd parties try to get the handshaking to work. I simply do not have any hours left in the day to do this. 16 a day is all I can manage and still have a little time for my children and some sleep once in a while.
That, with respect, is an MN management issue and they should resolve this.
Quote
We designed the communications in the Classic to compete with our nearest competitors and provide two plug-and-play interfaces: the Local Application and MyMidNite.
Ok I could say lots about this, just briefly, design for customers not competitors.
Not by any measure is the Localapp and Mymidnite within the definition of Plug and Play. Sorry, but they are both relatively complex bespoke programs needing significant effort to set up and get going.
Quote
If these are not useful to some customers then the weight of additional features unfortunately rests with you (unless you can pay us for the additional development time and resources or buy 1000+ units).
This is just another way of saying
If you are not satisfied with our software offerings then don't buy our productsQuote
We are open and willing to provide any features that we can fit into the schedule and will provide us with additional sales to compensate for the development time, but we can't spend a lot of additional time on features that won't improve our sales -- that makes us lose money which we really can't do at this point.
ok, this is meaningless to me. Maybe thats a company requirement but it just discourages any desire from users to think they can contribute to product improvements.
Quote
To add another TCP connection I would have to implement system-wide mutex locking. I would have to re-architect the modbus stack. In fact, I would have to do all the things that were deemed too time-consuming and unnecessary way back when we were initially designing the communications. So I'm afraid one connection is all that there will ever be on the Classic. Just like one Start button on Windows, or one App Store on iOS, or one screen on the iPhone.
Sorry I had forgotten that you had already told me in a previous message that you have written the Classic's TCP stack routines. That in itself just seems incredible to me as there are just so any available TCP stack software packages available (IPv4 IPv6, written in ANSI C and trimmable for embedded cpu systems).
So without mutual exclusion processing it would seem your Classic stack is barely minimum.
Without intending to detract from your efforts and expertise, it is a pity that the TCP stack was not implemented properly in the first place then we wouldn't have this ridiculous crippled TCP connection limit with the Classic.
The Classic has been with us for 4+ years now. In all that time the stack issues could never have been revisited and resolved?
Quote
That having been said, please, please, please keep suggesting features. You guys are awesome and have pushed the Classic along very well with many of the things you have thought of. We do value your opinions and suggestions, we just cannot possibly implement them all.
Sorry but this almost seems pointless now.
I can think of many many suggestions for improvements made by forum users but nearly all are ignored.
Errors and bugs do get attended to promptly, as you would expect, dead simple cosmetic changes may get attention but everything else, cosmetic or more serious, generally get nowhere.
If its an actively being worked on feature such as the SOC processing then suggestions meet with more success.
If its the local app then nothing.
NOTE: No offence or annoyance was intended towards anyone by this reply. I am not attacking anyone BUT just trying to speak plainly and honestly.
dgd
No, the interval is not a function of handshaking. Handshaking takes less than 100 ms. Incidentally, handshaking was also in place when the interval was set for 1 minute. This was during Beta testing and was never intended as the final design. The interval is directly related to the amount of data our server can handle along with the number of customers we are supporting.
You cannot change the interval anyway, even if you could redirect the data stream, I'm afraid. The call-in time is hard-coded into the Classic firmware.
The rs232 is 19200 baud. For a 256-byte modbus packet that is ~120ms. If you sample every 1 second you then have over 800ms to process the data. You can also process data offline as well.
Um... tcp headers and routing info (?) are never encrypted. Read up on TSL/SSL or IPSEC to get a better idea of how it works.
Incidentally our TCP stack is off-the-shelf. The one port is a design decision, not a technical limitation.
My technical documentation is up-to-date. To make it readable to fellow humans though it would need a drastic overhaul. Management has deemed this an extremely unnecessary task and do not want to release this IP. For academic information, however, here is an overview on how the handshake works:
http://en.wikipedia.org/wiki/Secure_Remote_Password_protocol
With all due respect, if we are so far off your mark why are you here, DGD? You are free to buy anything else on the market or make your own. You can probably still return your MidNite gear for a full refund. I cannot understand why someone who is so frustrated would continue this relationship?
I very much appreciate DGD input on here. With out that I would have no understanding of how all this works. To the credit of Midnite they are open in having this forum.
I have a friend who developed a product and debated if he should make it open source. His decision was to make it open . He found that the community did a lot of engineering on it and made it better which he could incorporate into the next version he put out. It didn't seem to make his product less popular or change his income.
I want DGD to keep digging into these things and taking the time to explain and comment on them. Until he just explained the data going out to mymidnite I had no idea some of the security issues that could be involved and had no idea that this was encrypted.
Please don't invite him to bug off !
Oh please don't get us wrong we are not asking anyone to bug off. We are simply trying to maintain a profitable business and develop new products the market needs. The real issue is the Classic is basically limited to what it can do on several fronts. We can design and build the ultimate charge controller but it will cost more. Maybe that's OK? Maybe we will sell thousands of them if we did? We just don't know.
What we are trying to do is work towards a line of Inverters and stuff and also Mario will be working on the Classic 2.0 So we are listening to all the suggestions and keeping an open mind. I still see some form of com box that is the do all be all device and it handles the Modbus server stuff so you can have 4 or 5 devices asking it stuff.
Ryan
Quote from: ClassicCrazy on March 21, 2014, 09:24:46 AM
I very much appreciate DGD input on here. With out that I would have no understanding of how all this works. To the credit of Midnite they are open in having this forum.
I have a friend who developed a product and debated if he should make it open source. His decision was to make it open . He found that the community did a lot of engineering on it and made it better which he could incorporate into the next version he put out. It didn't seem to make his product less popular or change his income.
I want DGD to keep digging into these things and taking the time to explain and comment on them. Until he just explained the data going out to mymidnite I had no idea some of the security issues that could be involved and had no idea that this was encrypted.
Please don't invite him to bug off !
Quote from: Halfcrazy on March 21, 2014, 12:00:12 PM
Oh please don't get us wrong we are not asking anyone to bug off. We are simply trying to maintain a profitable business and develop new products the market needs. .....
Ryan
Honestly, I never thought I would see the day. I put my Morningstar MPPT60 up for sale yesterday (to buy a Classic). This was my first piece of Solar hardware and is dear to my heart. But .. the Classic with the added value of Mymidnite and the WizBang Jnr have made it redundant. So I am all for enhancing a product with new features..
But there comes a time when its just not technically feasible and/or does not make business sense. Everywhere I go in this country, I see the Xantrex/Schneider XW and wonder WHEN will someone dethrone them from the king of the Inverter hill. I hope it will be Midnite Solar that does this, with the Classic II and a communications ecosystem to tie it all together. IMHO, I think the time is fast approaching when we will no longer see new features in the firmware, but only bug-fixes.
Look on the bright side guys .. We get to explain to the wives why we need new gear! lol :o 8)
Quote from: DMJ72 on March 21, 2014, 01:18:58 PM
Look on the bright side guys .. We get to explain to the wives why we need new gear! lol :o 8)
DMJ;
Eggs Ackly!
My old excuse of
"Honey, I don't have the bass boat, Cessna or muscle car I would love" has worn thin.
The only difference between the men and the boys is the price of their toys and all that!
Tom
Quote from: atop8918 on March 21, 2014, 04:20:24 AM
No, the interval is not a function of handshaking. Handshaking takes less than 100 ms. Incidentally, handshaking was also in place when the interval was set for 1 minute. This was during Beta testing and was never intended as the final design. The interval is directly related to the amount of data our server can handle along with the number of customers we are supporting.
You cannot change the interval anyway, even if you could redirect the data stream, I'm afraid. The call-in time is hard-coded into the Classic firmware.
The evidence is here that you were able to adjust the interval right down to one minute, this happened on one of my Classics. You then readjusted it back to 10 minutes and the firmware was never changed to achieved this changing from 10 minutes to 1minute and back to 10 minutes.
This I realised at the time was the server initiating comms between it and my Classic - a higher level handshaking between devices/programs rather than lower level data link handshaking.
The restrictions on interval and data storage would not exist if the link was made to a local network server rather than over the network to a very remote Mymidnite server.
dgd
Quote from: atop8918 on March 21, 2014, 05:43:30 AM
Um... tcp headers and routing info (?) are never encrypted. Read up on TSL/SSL or IPSEC to get a better idea of how it works.
I know how it works. The point I was making is that only the data is encrypted? Why? Its only Classic ID plus performance data, volts, amps etc. So what function does encrypting that serve. Or is it to somehow protect your version of higher level process handshaking and as such the encryption of user data is just a by product of that?
Quote
Incidentally our TCP stack is off-the-shelf. The one port is a design decision, not a technical limitation.
A commercial stack package without Mutex processing? Thats a new one for me as the very nature of a TCP stack would need mutual exclusion locking mechanisms as an integral part of its ability to function.
Quote
My technical documentation is up-to-date. To make it readable to fellow humans though it would need a drastic overhaul.
Apologies, I thought the purpose of docs, whether inline with source code or separate files/printed docs, was to convey meaning and explanation of coding decisions to other human beings and not just the original programmer. So this, as such, does not exist?
Quote
Management has deemed this an extremely unnecessary task and do not want to release this IP.
:o
Quote
For academic information, however, here is an overview on how the handshake works:
http://en.wikipedia.org/wiki/Secure_Remote_Password_protocol
Thanks for that. However, my academic background in computer science has provided me with a very sound understanding of how handshaking works and can be implemented (amongst many other computing issues)
Quote
With all due respect, if we are so far off your mark why are you here, DGD? You are free to buy anything else on the market or make your own. You can probably still return your MidNite gear for a full refund. I cannot understand why someone who is so frustrated would continue this relationship?
Wonderful, thanks for that.
I have reiterated several times in this forum that I am very happy with my Midnite equipment and have even stated that it is likely the best there is available. I have not commented personally on anyone and have tried to address the issues rather than comment on the people concerned.
Please remember that this thread originally started with a simple request to make an encrypted data stream available in an unencrypted form to the owners of said data.
In response to this you have said: can't do this, won't do this, management won't allow it, don't have time, too complicated, can't disclose company secrets, you don't need it, alternatives exist, the coding is complicated, won't understand documentation (or it doesn't exist), can't support user questions if info released, will never change and finally
if you are unhappy with these reasons then don't use Midnite Such a varied set of blockers, well done. :)
Ok, enough said. Thanks for your efforts and continued good work A (and Midnite)
dgd
[and sincere apologies to anyone who may be upset or offended by my posting - that was never my intention]
Quote from: Halfcrazy on March 19, 2014, 05:40:18 AM
You guys do realize there are unlimited RS232 connections available?
Quote from: atop8918 on March 21, 2014, 04:21:58 AM
The rs232 is 19200 baud. For a 256-byte modbus packet that is ~120ms. If you sample every 1 second you then have over 800ms to process the data. You can also process data offline as well.
I would appreciate it if you can please explain how to get the whole register range in less than a second over serial. Ive tried and failed here:
http://midniteforum.com/index.php?topic=1751.0
Thing is at faster sample rates, we only need a dozen registers. At slower rates, maybe 30 or 40. But USB mode has to be one or other. I must be missing something.
Do you need to use one of the follow me ports?
Quote from: zoneblue on March 22, 2014, 12:09:47 AM
Do you need to use one of the follow me ports?
Any one (or more) of the 3 RS-232 jacks.
Ok thanks bob. Are those ports fairly robust, and its midnite "policy" to allow slash recomend their use for monitoring purposes, or more of a case, be careful and you are on your own if you blow it up?
Quote from: zoneblue on March 22, 2014, 05:49:21 PM
Ok thanks bob. Are those ports fairly robust, and its midnite "policy" to allow slash recomend their use for monitoring purposes, or more of a case, be careful and you are on your own if you blow it up?
I have never had any issues with these ports, they certainly seem robust and I have not seen one of those random resets associated with rs232 port usage. There is one I seem to remember also brings 9v power out on one pin.
I tried using follow-me without the loopback connection (3 Classics) so I could use a serial port but follow-me needs all of the ports (unless that has changed in later firmware :) )
Have often thought how useful it would be if a PC could run the local app that connected to the Classic via one of those serial ports.
dgd
Thanks. Where did you find documentation for them? Pinouts etc?
Page 79 of Classic manual - see pdf in documentation section
Pinouts are at the very end of the Classic modbus document. (needs an update for the WB Jr. stuff)
These connectors can access the same stuff as the modbus over TCP/IP.
The main difference is that the RS-232 connections need a modbus ID address for
the Classic, which is contained in the very first modbus byte. Default is 10 (decimal).
For modbus over TCP/IP, each Classic must have its own IP address and/or port.
boB
Ok, ta. And you guys are ok with the RJ11s being used for monitoring?
But anyway if followme requires two, and the MNGP/LP requires one. Im just seeing that we are all round best off using the ethernet port. Im ok about that, its working totally stable, so long as you maintain an open connection, and keep it open, just like the local app does.
BTW ive roughed up a wiki page about the com ports here:
http://code.google.com/p/theblackboxproject/wiki/ClassicComPorts
Any changes let me know.
Quote from: zoneblue on March 23, 2014, 04:09:03 PM
But anyway if followme requires two, and the MNGP/LP requires one. Im just seeing that we are all round best off using the ethernet port. Im ok about that, its working totally stable, so long as you maintain an open connection, and keep it open, just like the local app does.
Or have the follow me loop back done in one rs232 port.
Probably 90% of classics are alone, no follow me necessary
Maybe 9% is two classics so just one bidirectional rs232 port needed on each for follow me
Three classics or moe connected to same battery bank would need to use two rs232 ports on classics in middle, or use a rj11double socket adaptor for middle classics.
That should make a free rj11 rs232 port on each classic for com box connection ;)
...just another idea :)
Dgd
I am not very smart with RS232 but I "Think" if the classics are wired together you can talk to all of them through 1 RS232 cable.
Hopefully, eventually we can get Follow Me to work using only bi-directional RS232 cable
for multiple (more than two) Classics.
Our modbus is MODified... What I mean is that normal modbus does not echo or forward
packets through to other devices. We do this now but it can get bogged down if not
careful. Follow me using the loop-around works pretty well and a missed packet here
or there doesn't matter. Flawless communications is my goal of course but that
just doesn't happen. I'll settle for close to flawless.
I do like the idea of focusing on new product lines. There is no satisfactory resolution to current discussion thread because original design was for proprietary but free bundled tools to monitor the Classic.
But for the next seminal MN product, it would be a great idea to include right in the design phase some API's that can be used by all developers to write software to communicate with the device. These API's ideally should be RESTful API's that do not requires stateful processing and allow multi-access. For critical comms, standard queueing solutions such as rabbitmq could be used instead of prorietary SCADA protocols.
And a lot of the MN engineers seem to have a lot of depth in software development already, so perhaps their personal development plans could include going to some conferences or reading some O"Reilly books on learning how to to write good APIs and distributed computing.
Quote from: Halfcrazy on March 21, 2014, 12:00:12 PM
Oh please don't get us wrong we are not asking anyone to bug off. We are simply trying to maintain a profitable business and develop new products the market needs. The real issue is the Classic is basically limited to what it can do on several fronts. We can design and build the ultimate charge controller but it will cost more. Maybe that's OK? Maybe we will sell thousands of them if we did? We just don't know.
What we are trying to do is work towards a line of Inverters and stuff and also Mario will be working on the Classic 2.0 So we are listening to all the suggestions and keeping an open mind. I still see some form of com box that is the do all be all device and it handles the Modbus server stuff so you can have 4 or 5 devices asking it stuff.
Ryan