Strange reset during the day

[aroxburgh]

boB:
Some thoughts on the Classic firmware...

We all would not be so concerned about the random resets if their impact were less (although of course the cause or causes needs to be found, and fixed, to reduce the possibility, down the road, of further unintended side-effects). 

Right now it would appear that the Classic has no clearly defined cold start and warm start states.

A cold start is defined as power-on reset after having sat powered down long enough for all RAM locations to decay below logic thresholds; all variables are initialized to default values or values based on flash/EEPROM storage.

In a warm start, the Classic would use a special mechanism to determine (with a high measure of probability) that a reset has occurred, but was not caused by a power-up from cold. A warm-start state would allow key variables to be restored to or remain at the value they had prior to the reset. In particular I am thinking of kWh and Ah. I expect that there are others too. This would typically be achieved by having the reset handler code write a special key (unlikely bit pattern) to one or more RAM locations, but not before the handler looks in those locations to see if the key values have already been written. If they have been, then we set the warm-start flag. We then initialize all relevant variables using conditional initialization (the condition is: NOT warm_start). Those variables that are normally initialized by writing them from literal values stored in the firmware code (as opposed to restoring the values from flash/EEPROM) are left "as is," perhaps with a test to ensure that they are still in-range. Some variables, e.g., the clock time, are already preserved across reset, but that may be merely due to reading the time from a battery-backed-up time chip.

In summary, I'd like to see you define a warm-start state, so that in the event of the kind of resets we are seeing at the moment, the parameters that are most important, namely kWh and Ah can be relied on, as we monitor the energy performance of our renewable energy systems. Of course, probably like many other users, I too would like to see remote monitoring (including password-protected system configuration) using a web browser, voice alerts for conditions such as low battery, and improved presentation of the performance results, but am willing to wait for these, if there could be some small additions to the code to make it more practically useful and reliable where it counts.

It is good, as several have pointed out, that Classic energy generation continues largely unabated during the random resets, but at the end of the day it is vital to know how many kWh were generated. A big part of energy management and planning is knowing how big your energy resource is! Otherwise it's little better than touch wood and hope for the best. 

Al
aj4rf

[boB]

boB:
Some thoughts on the Classic firmware...

We all would not be so concerned about the random resets if their impact were less (although of course the cause or causes needs to be found, and fixed, to reduce the possibility, down the road, of further unintended side-effects). 

Right now it would appear that the Classic has no clearly defined cold start and warm start states.

A cold start is defined as power-on reset after having sat powered down long enough for all RAM locations to decay below logic thresholds; all variables are initialized to default values or values based on flash/EEPROM storage.

In a warm start, the Classic would use a special mechanism to determine (with a high measure of probability) that a reset has occurred, but was not caused by a power-up from cold. A warm-start state would allow key variables to be restored to or remain at the value they had prior to the reset. In particular I am thinking of kWh and Ah. I expect that there are others too. This would typically be achieved by having the reset handler code write a special key (unlikely bit pattern) to one or more RAM locations, but not before the handler looks in those locations to see if the key values have already been written. If they have been, then we set the warm-start flag. We then initialize all relevant variables using conditional initialization (the condition is: NOT warm_start). Those variables that are normally initialized by writing them from literal values stored in the firmware code (as opposed to restoring the values from flash/EEPROM) are left "as is," perhaps with a test to ensure that they are still in-range. Some variables, e.g., the clock time, are already preserved across reset, but that may be merely due to reading the time from a battery-backed-up time chip.

In summary, I'd like to see you define a warm-start state, so that in the event of the kind of resets we are seeing at the moment, the parameters that are most important, namely kWh and Ah can be relied on, as we monitor the energy performance of our renewable energy systems. Of course, probably like many other users, I too would like to see remote monitoring (including password-protected system configuration) using a web browser, voice alerts for conditions such as low battery, and improved presentation of the performance results, but am willing to wait for these, if there could be some small additions to the code to make it more practically useful and reliable where it counts.

It is good, as several have pointed out, that Classic energy generation continues largely unabated during the random resets, but at the end of the day it is vital to know how many kWh were generated. A big part of energy management and planning is knowing how big your energy resource is! Otherwise it's little better than touch wood and hope for the best. 

Al
aj4rf

I like the idea of a "warm start" as you say and that COULD be done of course.  The only issue that I
have with that is, what if the variables are corrupted ?  Then, you would warm boot with corrupted
states and numbers.  At this point, we would, as  you say, have a high confidence that they
would be in good shape.

I'd rather just get the resets fixed so we don't have to worry about that.  I suppose that warm
boot mode could be selectable so if it is only network problems causing resets as it looks like
right now, then this could be usable.   Normally this would be disabled of course.

If we don't figure this weird situation out soon, I may just do something like you are proposing.

BTW, did you guys get the newest and latest code for the My Midnite calling in fix ??

http://fusion.midnitesolar.com/MidNiteSolarSetup_6-6-2013_v3.72.exe

boB

[dogfox]

Hi All,

Just want to put my two bits in..

I have a midnite 200 that is currently regulating my (new) 500w 48v turbine for a 24v 1600Ah battery bank in my rops setup. 

The mfg says the turbine should be kept in a voltage range between 48 and 150.

Yesterday, we (finally) got good wind (40-50 kph, ~25 to 30mph) to make up for lack of sun and, in the process of checking how the turbine was performing, I noticed that the controller reset - several times.  In the time the controller takes to get back up and regulating, the voltage got up to, at one point, 184v.  I know this because I have a hand built, arduino based logger that logs stats on my whole setup.

Now, I appreciate those with midnites regulating their pvs saying the resetting is not that big a deal because the controller is still 'making power', but please spare a thought or two for those of us with turbines that MUST be loaded..  I hate to be demanding but we need to have a solution (or workaround) found now.

I am sure you guys can appreciate that my better half is not best pleased that I am turning our only source of generation off because I cannot trust the controller (that *I* told her was the greatest thing since sliced bread)

Some details:
The controller is a 200v rev 4, about two months old (It replaced another that failed)
It is on a network with a static ip.
It *seems* to reset when there is a wild swing in current being delivered to the batts.
I don't run the local app all day.
It *can* have web access but I am on a limited data plan so I cannot be sending and receiving lots of data (should someone need to remote troubleshoot).

Thanks for reading and please advise

[boB]

Hi All,

Just want to put my two bits in..

I have a midnite 200 that is currently regulating my (new) 500w 48v turbine for a 24v 1600Ah battery bank in my rops setup. 

The mfg says the turbine should be kept in a voltage range between 48 and 150.

Yesterday, we (finally) got good wind (40-50 kph, ~25 to 30mph) to make up for lack of sun and, in the process of checking how the turbine was performing, I noticed that the controller reset - several times.  In the time the controller takes to get back up and regulating, the voltage got up to, at one point, 184v.  I know this because I have a hand built, arduino based logger that logs stats on my whole setup.

Now, I appreciate those with midnites regulating their pvs saying the resetting is not that big a deal because the controller is still 'making power', but please spare a thought or two for those of us with turbines that MUST be loaded..  I hate to be demanding but we need to have a solution (or workaround) found now.

I am sure you guys can appreciate that my better half is not best pleased that I am turning our only source of generation off because I cannot trust the controller (that *I* told her was the greatest thing since sliced bread)

Some details:
The controller is a 200v rev 4, about two months old (It replaced another that failed)
It is on a network with a static ip.
It *seems* to reset when there is a wild swing in current being delivered to the batts.
I don't run the local app all day.
It *can* have web access but I am on a limited data plan so I cannot be sending and receiving lots of data (should someone need to remote troubleshoot).

Thanks for reading and please advise

dogfox,  do you have a clipper installed on your turbine ?

You did not mention if you had a dump load or not.

How did the first controller fail ?  Was it overvoltaged ?

A Clipper is necessary to keep the turbine loaded even if the Classic
is on and running because when the batteries get full or exceeds
the capacity of the power of the turbine, the turbine will be unloaded
and the turbine will rise very high.   184V will not hurt the Classic but
the Classic by itself will not load the turbine down with a battery
connected in high winds.

A clipper will also load the turbine at a fail-safe voltage even if the
Classic is off, reset or disconnected.

Thanks !
boB

[boB]

dogfox,  Another question...

What kind of power or current did you see during the current swings when you saw it reset ?

Thanks,
boB

[boB]

BTW folks....

The newest firmware...

http://fusion.midnitesolar.com/MidNiteSolarSetup_6-6-2013_v3.72.exe

has the data abort information registers in them.

If your Classic has done a reset, check these registers if you have modbus reading capability.
Otherwise, PM or email me.

Registers   4340 through 4343   (addresses  4339  through 4342)

Those registers should contain the information we need to help track this problem
down due to the watch-dog resets.

Thanks,
boB

[dogfox]

dogfox,  do you have a clipper installed on your turbine ?

No, I don't.

You did not mention if you had a dump load or not.

Yes, I have a dump load capable of dissipating 2000w.

How did the first controller fail ?  Was it overvoltaged ?

Not that I am aware.  Unfortunately, I cannot say for sure.  I had lowered the old turbine (24v 500w) at the start of the summer as I don't need the added wind power then.  At the end of the summer I raised the old turbine and, at the next moderate wind event, I noticed that the controller was putting out NO current.. Everything else was fine (was accepting firmwares, display looked correct, voltage was accurate, etc).   I was most upset when the realization dawned that the controller died and I had no idea when, nor why.

A Clipper is necessary to keep the turbine loaded even if the Classic
is on and running because when the batteries get full or exceeds
the capacity of the power of the turbine, the turbine will be unloaded
and the turbine will rise very high.

I have been eyeing the clipper but, once my arduino logger is fully operational, it will be doing fail safe braking with pwm.

   184V will not hurt the Classic but
the Classic by itself will not load the turbine down with a battery
connected in high winds.

With all due respect, I think you missed my point.  The turbine should not rise past 150v, and it did.  I did because the controller was resetting.  When the controller *was* operating properly, the turbine was being held at approx 18 amps on the curve I'm currently playing with (which relates to approx 100v).  I saw gusts push the amps up to 30 but that was still only 105v and a bit.  My curve's last two points are 30A 105v and 48A 106v. 

A clipper will also load the turbine at a fail-safe voltage even if the
Classic is off, reset or disconnected.

I do understand the purpose of the clipper, I'm just disappointed that I'll need to purchase another midnite product as a backup to my existing midnite product.

[boB]

dogfox,  do you have a clipper installed on your turbine ?

No, I don't.

You did not mention if you had a dump load or not.

Yes, I have a dump load capable of dissipating 2000w.

How did the first controller fail ?  Was it overvoltaged ?

Not that I am aware.  Unfortunately, I cannot say for sure.  I had lowered the old turbine (24v 500w) at the start of the summer as I don't need the added wind power then.  At the end of the summer I raised the old turbine and, at the next moderate wind event, I noticed that the controller was putting out NO current.. Everything else was fine (was accepting firmwares, display looked correct, voltage was accurate, etc).   I was most upset when the realization dawned that the controller died and I had no idea when, nor why.

A Clipper is necessary to keep the turbine loaded even if the Classic
is on and running because when the batteries get full or exceeds
the capacity of the power of the turbine, the turbine will be unloaded
and the turbine will rise very high.

I have been eyeing the clipper but, once my arduino logger is fully operational, it will be doing fail safe braking with pwm.

   184V will not hurt the Classic but
the Classic by itself will not load the turbine down with a battery
connected in high winds.

With all due respect, I think you missed my point.  The turbine should not rise past 150v, and it did.  I did because the controller was resetting.  When the controller *was* operating properly, the turbine was being held at approx 18 amps on the curve I'm currently playing with (which relates to approx 100v).  I saw gusts push the amps up to 30 but that was still only 105v and a bit.  My curve's last two points are 30A 105v and 48A 106v. 

A clipper will also load the turbine at a fail-safe voltage even if the
Classic is off, reset or disconnected.

I do understand the purpose of the clipper, I'm just disappointed that I'll need to purchase another midnite product as a backup to my existing midnite product.

Is your 2 kW dump load on the battery side of the controller ?  If so, there is no way that the controller can keep the
turbine below 150 volts if there is enough wind to put the controller into current limit, which can happen. When that
happens, the controller will raise its input (turbine) voltage to keep its output current at current limit or, if the
battery side dump load cannot keep the battery voltage down, the CC will raise the input (turbine) voltage
to keep the battery voltage at the Absorb or Float set-point.

Now, if there is always someone there to bring down the turbine during an upcoming storm, then you
won't have that problem but that's not very automatic.

Once the wind gets high enough to push the output above the highest power curve point ( 48A 106v ?),
it can still go higher in voltage and output current.

Still, the resets are not wanted and in normal wind conditions, it will unload the turbine while it re-boots.

What is your PWM braking technique you are working on with the Arduino ?  Will it PWM the turbine
on the turbine side ?  If so, will it do it on the AC side of the turbine rectifier ?  That is the best way
as then you don't need to add a diode between the rectifier DC output and the controller input
terminals.  That diode is to keep the input terminals on the controller from driving the dump
load on the input from power coming through the controller from the battery.

I am wondering why your Classic tended to reset when you saw high current swings ??

Were the multiple resets at ~approximately~ a minute between each other ??

Was there higher power each time that you could tell or did the high current
surge maybe just appear to start the string of resets ?

The time that I saw the reset happen in front of me with my home router, there was
very little or no power running through the Classic but once it started resetting,
it was kind of like sneezing...  It happened multiple times in a row at around
1 minute intervals.

boB

[dgd]

dogfox,
I have watched my Classic 250 reset when the turbine was making power in a blustery rainstorm. No Clipper but I did have a 3phase dump load in use. This load was online when the reset occured and the Classic restarted. The turbine voltage rose to about 135 before the Classic re-enabled the dump load.
Just FYI the load was six 2ohm 300W resistors from Missouri wind and solar, 3 sets of two in series. Each set was connected to a live AC output from the turbine, the other ends to the A1 A2 and A3 inputs oof a crydom D53PH25D SSR, the B1, B2 and B3 all connected together. The two control inputs connected to the Classic AUX1. Aux1 set PV HIGH voltage 110v
This voltage was the max I wanted to see my 48V turbine make.
This voltage limiter worked well.  I never had a runaway turbine. All up cost was about $180

dgd

[dogfox]

What an in-depth response!

So many questions.. I'll tackle them one at a time.

Dump load is on turbine side of a 3 phase rectifier (200A).  Dump load is switched on with 3-32v dc to three 50A SSRs turned on currently by one of the auxs on the midnite.  Each SSR connects to a phase of the turbine.  Dumpload is three 10ohm resistors wired in a delta config, one phase to each point.  DC output of the rectifier is then fed into the midnite.

The turbine *should not* ever get to the 48A point as that would be ~1300 watts from a 500w turbine and it *should* have furled by then.  But, if a serious storm (winds expected above 80kph) is forcast, I do lower the turbine for safety sake.  Lowering takes about 5 mins so not that big a deal. I do have to be home tho..

The arduino will eventually turn on the dumpload ssrs on under certain conditions (soc > 99.9, batt volts > 30, turbine volts > 120, turbine amps> 25) and use pwm to slow the turbine using more and more load until it is nearly stopped. 

I am wondering why your Classic tended to reset when you saw high current swings ??

So am I 

Were the multiple resets at ~approximately~ a minute between each other ??

Not that I saw.   

I went into the local app to try and export logs to see exactly what happened.  The logs showed one particular reset where "Kwh" went from .1 to 0 but that .1 was not added to "total kwh".  Timestamps were still 2 secs apart even tho "Battery Current", "Watts", "kWh", "charge state", and "Input current" all were zero (must have been resetting?).  During that particular time, input volts went to a high of 123.6. 

The next reset was 2 mins 2 secs later.  Watts were not particularly high for each (according to the log).  What was interesting is for each even the fet temp went DOWN and kept going down but the PCB temp went UP briefly then went down. During this reset, input volts went to a high of 165.7.

Also according to the logs, there are several places where "kWh" go from a value, say 2.2, in one row, to 0 on the next row with no corresponding increase in "total kwh"..  So those must be resets as well. 

I am wondering if the logs are not a bit suspect tho as there was a bit of the log where the timestamps went from 6/6/2013 5:19:03 then 6/6/2013 11:53:22 then 6/5/2013 21:20:21  on three consecutive rows and the values for each row were clearly from different times. 

I may be grasping at straws with some or all of this...

Was there higher power each time that you could tell or did the high current
surge maybe just appear to start the string of resets ?

Didn't seem like a string of resets.  Now looking at the logs, it might not have even been high current triggered.

Baffling.


I just want to get out there that I love the product and can see great potential in it.  However, this particular issue is a real showstopper for me as, for now, I cannot trust the CC nor any of the data coming out of it. 

I can attach csvs of the logs if you want
Cheers

[dogfox]

dogfox,
I have watched my Classic 250 reset when the turbine was making power in a blustery rainstorm. No Clipper but I did have a 3phase dump load in use. This load was online when the reset occured and the Classic restarted. The turbine voltage rose to about 135 before the Classic re-enabled the dump load.
Just FYI the load was six 2ohm 300W resistors from Missouri wind and solar, 3 sets of two in series. Each set was connected to a live AC output from the turbine, the other ends to the A1 A2 and A3 inputs oof a crydom D53PH25D SSR, the B1, B2 and B3 all connected together. The two control inputs connected to the Classic AUX1. Aux1 set PV HIGH voltage 110v
This voltage was the max I wanted to see my 48V turbine make.
This voltage limiter worked well.  I never had a runaway turbine. All up cost was about $180

dgd

A very similar setup to mine, with exactly the same aux1 setup except the voltage set point for me was 104.  Problem was, while the Classic is rebooting, aux1 is off.. 

[dgd]

The newest firmware...

http://fusion.midnitesolar.com/MidNiteSolarSetup_6-6-2013_v3.72.exe

has the data abort information registers in them.

If your Classic has done a reset, check these registers if you have modbus reading capability.
Otherwise, PM or email me.

Registers   4340 through 4343   (addresses  4339  through 4342)

Those registers should contain the information we need to help track this problem
down due to the watch-dog resets.

Thanks for that update, installed and awaiting next reset,although no doubt Sods law will men no reset for ages.. 
just wishing for those date and time settable registers now 

dgd

[dgd]

...
This voltage limiter worked well.  I never had a runaway turbine. All up cost was about $180

dgd

A very similar setup to mine, with exactly the same aux1 setup except the voltage set point for me was 104.  Problem was, while the Classic is rebooting, aux1 is off..

yes of course, I was thinking of AUX1 PV High Hold time but a reset just disables AUX while rebooting.
I suppose the way would be a little circuit with a latch and timer to hold relay on for 20 seconds     then again removing  the bug is what we need.
But as boB says the solution to protecting the turbine is a Clipper, albeit somewhat expansive. Please share your PWM progressive load system when you get it designed, I'm sure there would be a lot of interest in it   

dgd

[RossW]

Dump load is on turbine side of a 3 phase rectifier (200A).  Dump load is switched on with 3-32v dc to three 50A SSRs turned on currently by one of the auxs on the midnite.  Each SSR connects to a phase of the turbine.  Dumpload is three 10ohm resistors wired in a delta config, one phase to each point.  DC output of the rectifier is then fed into the midnite.

Just a small comment if I may. When switching a dumpload across a 3-phase circuit in either star or delta, you only need *TWO* relays. You could pull that third one out and keep it for a spare or something

[RossW]

The newest firmware...
...
has the data abort information registers in them.

Bob, talking to a couple of guys, the new firmware appears to do something bad to the ethernet comms.
Modbus reads that used to be virtually instant are refusing connections for anything up to around 30 seconds.