More robust ethernet communication

[thooker]

OK, I sent an email detailing my setup.  In the interest of open communication, I have included it here.

Hi Ryan,
This is Tracy, and I am on the Midnite forum.  I was asked to send you all the info I can on my ethernet crashes.

Frankly, lately it is like I have been walking on egg shells when I communicate with the Classic.  I am very careful to not ask it 'too much'.  I always tried to wait several minutes between turning on the local app.

My system is running a mountain communication site, which feeds a town of ~10,000 residents with internet.  The wireless ISP uses my site to make the jump to the other town.  A couple of days ago we had a cold snap and lots of clouds, so the battery voltage went a little below what the inverter could support.  We lost AC power from the inverter, which took down their backhaul link.  When the backhaul went off, the load on the batteries was lessened, which allowed the voltage to go up, and the inverter came back on.  The ISP said that it was about a 2 minute on / 2 minute off cycle, which lasted for several hours before we could make the trip to the mountain top.

I have auto reset turned on, so it reset the charge controller at midnight, however, the up/down of the inverter was still going on at that time.  Around 1 AM I reset the charge controller by turning it off, then back on.  When I got back to my house at 4:30 AM, I tried the local app, and it did not connect.  I tried to connect using the local app AND the mymidnite.com site throught the next day, and got nothing.  I could ping the port correctly.

The following day I tried again using the PC local app, and it worked.  Apparently the CC reset itself at midnight, and all was well.  I noticed that the local app background was yellow, due to the clock being different from the PC clock.  I dont remember the exact order, but within 1 minute I started and stopped, and restarted the local app on my PC, and from that point on, I had no contact with the Classic again.  I tried on my android phone, same result.  I looked at the mymidnite.com, and it was no longer updating.  I could still ping the Classic.

I had to wait for it to reset again at midnight, which it did.  The next morning (which was Jan 6) it was OK again.  I have just tried to use the local app on both my PC and on android, in various combinations, to attempt to break the communications.  It, of course, is working fine!  I can not get it to break now.  However, that does not mean that it is fixed!

I am pinging the port, and I get returns between 30 ms and 80 ms.  My configuration is the following:

Both the site with the Classic and my home are serviced by the wireless ISP.  At the site with the Classic, the Classic (static IP) is connected to a switch, then to the backbone radio  5.8 GHz link.  It goes about 11 miles to the WISP headquarters.  From there it goes to the internet, and also spreads out in a star configuration.  One of those spokes goes about 5 miles to a tower, which then goes about 2 miles to my house via a Ubiquity RF link.  At my house I have a Linksys WRT54G wireless router (static IP), which connects to my PC (Dynamic IP).  Please see attached drawing.

Firmware:  Classic 1849, Net 1839.  S/N CL246.  Local App version 0.3.61.

If there is anything else I can do, please let me know.  It is VERY important to me to be able to see what the unit is doing.  I have been (I believe) very patient with this process, but I dont like emergency trips to the mountain at 11PM!

[offgridQLD]

A good sighn the little 3v lithium coin cell-watch battery size (1216) needs replacing in the classics display unit is when you get a yellow local app screen after a power cycle of the classic. As it looses it's date and time setting.Often overlooked preventative mantanance . I have mine down in the calander now for a watch battery change every 12 months.

Not the answer to the network issues but will help eliminate one variable.

Kurt

[boB]

A good sighn the little 3v lithium coin cell-watch battery size (1216) needs replacing in the classics display unit is when you get a yellow local app screen after a power cycle of the classic. As it looses it's date and time setting.Often overlooked preventative mantanance . I have mine down in the calander now for a watch battery change every 12 months.

Not the answer to the network issues but will help eliminate one variable.

Kurt

Realize that the midnight auto-restart does not power-off and power-on the classic so the clock and calendar
will stay the same.  Not sure what to say if the time/date changes after a reboot without power to the
Classic/MNGP going off and back on again ?  That would be pretty weird but anything can happen.

Also, OffgridQLD, which inverter do you have with Ethernet ?

boB

[offgridQLD]

All I know is I was getting lots of (yellow local app) time out of whack . Every time I would reset the time again it wouldn't last. So I went and replaced both the clock batterys in my two classics and Its been fine.

I was getting the yellow local app at times without fully power cycling the classics on the breaker. Im not saying it was happening after a 12pm reboot but it was happening some time through the day .....why how I diddn't investigate just changed the two batterys that were both low voltage- dead and its been all good. So I can only guess for some reason i was getting a power cycle of the classic over the day. The rest of the system never shut down so the classics always had DC power to them.


The inverter I have is a locally made.... (Australia) Selectronic PS1 6000w 48v inverter/charger. I just have it connected to my local network so I can dial in to its monotoring/logging software remotely. 

Kurt

 

[offgridQLD]

In the name of getting to the bottom of this (or just prove I am not crazy

I took a small 2 min video and uploaded it on my you tube channel. I logged into my (house) classic 150 that has the WBJR on it using the android monitoring app. The system starts out in float around 11AM everything happy I then switch the screen to show the WBJR data and shortly after I get a small dropout in communication that quickly recovers (not so concerned about thisfast dropout & recovery but thought I would mention it anyhow)

Then I go to the (offline data screen) and within a second or two it loads the calender view with the offline data for each day. Then BINGO! you can see in the top left hand corner of the screen my classic is showing status is showing (Resting) then absorb and shortly after float . So by me going to view the (offline data) has triggered the full reset my classic. Because of this reset the classic cycles back to resting/absorb/float like its starting a new day. I then go to the total KWh for the day and it's been reset back to 0kwh

Like others have mentioned it doesn't happen 100% of the time but I got lucky this time while recording it and it did it first go for me. It's the same when using the (local app by midnite) on any platform. Going to the offline logs isn't the only way to trigger it but it seems like a good/reliable way to upset it.

So I use the local app very tentatively not to upset/reset my charge controllers.

I hope this helps in some way .

Anyhow link to the video showing what I explained above....https://www.youtube.com/watch?feature=player_detailpage&v=2DokCstXyQY

Kurt

[Halfcrazy]

So here is a big question. Is everyone that is having this issue using My Midnite also? Andrew has found a couple things already and they relate to My Midnite, Well they relate to the fact that if the Local app and My Midnite request data at the same time it overloads the memory

[zoneblue]

ok, interesting. Reading some of the folk above, there does seem to be a thread of 'first i tried local app on pc, then android, then mymidnite'. You have to wonder how about all those opens and closes and possible overlaps.

Since ive been using the single connection blackbox (only, use local app maybe 3 or 4 times a year for config), i havent had a single ethernet issue. My only observations are that in general modbus reads over ethernet are slightly slower than the previous firmware, and that the classic self reboots (lots) more often than the previous.

However seeing as Andrew is having a push on this i will reconfigure blackbox back to 1 minute, open, read  and close. That setup would previously show up any issues within 14 hours.

[2twisty]

I am using MyMidnite now, but had the problem before I started using it.

[Resthome]

So here is a big question. Is everyone that is having this issue using My Midnite also? Andrew has found a couple things already and they relate to My Midnite, Well they relate to the fact that if the Local app and My Midnite request data at the same time it overloads the memory

No MyMidNite in my configurations. So that isn't my disconnect issue.

[Resthome]

Might want to note the comments by Graham on TCP traffic here.

http://midniteforum.com/index.php?topic=2136.msg21395#msg21395

[Graham]

Might want to note the comments by Graham on TCP traffic here.

http://midniteforum.com/index.php?topic=2136.msg21395#msg21395

Just want to mention a couple of things about my android app related to Kurt's video

when you slide over to the day log calendar, you are not triggering a read request for the logs from the classic at that point, the app has a background service that polls the classic and broadcasts the information to the rest of the app, each tab has a listener that receives the info related to itself.
+ the app maintains  (up to) 3 tabs or views in memory at any time, the viewed tab and the one to the left and right, so when you slide over to the calendar, you are actually loading the chart etc.
the charts take longer to load than other views (the hour chart has 60 X 24 minute entries to load for 5 arrays of values) , that takes some cpu time away from the modbus service and that's when I see comms errors in wireshark.

[Resthome]

In the name of getting to the bottom of this (or just prove I am not crazy

Then I go to the (offline data screen) and within a second or two it loads the calender view with the offline data for each day. Then BINGO! you can see in the top left hand corner of the screen my classic is showing status is showing (Resting) then absorb and shortly after float . So by me going to view the (offline data) has triggered the full reset my classic. Because of this reset the classic cycles back to resting/absorb/float like its starting a new day. I then go to the total KWh for the day and it's been reset back to 0kwh

Like others have mentioned it doesn't happen 100% of the time but I got lucky this time while recording it and it did it first go for me. It's the same when using the (local app by midnite) on any platform. Going to the offline logs isn't the only way to trigger it but it seems like a good/reliable way to upset it.

So I use the local app very tentatively not to upset/reset my charge controllers.

I hope this helps in some way .

Anyhow link to the video showing what I explained above....https://www.youtube.com/watch?feature=player_detailpage&v=2DokCstXyQY

Kurt

Kurt,  are the MNGP in the Daily Log display window by any chance?  Having the Classic MNGP in this window and doing and OFFLINE DATA export from the LA will reboot the Classic 100% of the time. It's a know bug. The RFR code is 104 I believe which is the Watchdog Timer reset.  You might want to record and post the RFR code here the next time it Reboots  like that. Also there were some Debug register on one of the LA status pages you might want to also post. They were suppose to help troubleshoot these reboots. That is if they ever got set to be the correct registers. boB, Andrew or Ryan will have to verify if that is still useful information.

[boB]

Good news, sorta...  Andrew has duplicated the problem and knows what is happening.

It's a packet memory re-allocation problem.  When this happens, the Classic can not fill
the requested packet with data but it can ping, etc. since that doesn't require packet memory.

So, we are well on our way to finally getting this fixed.  Not sure exactly when but this
is the highest priority at the moment so sooner rather than later I expect.

boB

[offgridQLD]

John,
        I don't think that would be my issue as both My classics  MNGP are just in the home window or the screen they default to  (I guess thats what you could call it) I  avoid using the MNGP. As they are with the two classics out in the power room away from the house with the snakes and spiders . All l monitoring and control and tweaking is over the the network.

Bob thats great news. Just being able to point to the problem is 99% of the challenge well done.

Kurt

[Resthome]

Good news, sorta...  Andrew has duplicated the problem and knows what is happening.

It's a packet memory re-allocation problem.  When this happens, the Classic can not fill
the requested packet with data but it can ping, etc. since that doesn't require packet memory.

So, we are well on our way to finally getting this fixed.  Not sure exactly when but this
is the highest priority at the moment so sooner rather than later I expect.

boB

boB, is this going to fix just the disconnects we have been seeing or is it going to fix the reset/reboot issue Kurt showed in his video also?