System shutdown...batteries were at 56% and went to 0%

[RightRudder]

Good Evening,

Newbie and DIY learn as I go here, so go easy!

I am totally off grid, have a MN15-12KW-AIO and (4) MN PowerFlo5 batteries.  Our system has been working and just (2) nights ago it went totally dark.  Batteries went from 56% to 0% in the blink of an eye.  I have been in contact with Rick who has been a huge help.  He is working with me on getting firmware updates completed. 

In trying to charge the batteries back up, it has been painfully slow.  We have very little solar input due to the weather, and the generator is only putting in .018KW.  The generator is a Honda EU6500is.  It kicks up when the solar input increases, but when it disappears it drops off again.  I have gone from 0% to 8% in (2) days on the batteries.  The generator worked just fine prior to this issue charging the batteries.  The generator will still run everything else, but it is very limited on the input to the batteries. Does anyone have any thoughts on how to make this go faster?  Rick had helped me try to trick the batteries with changing settings and that didn't work. I am just throwing a bone out there as Rick tried all the tricks he had. 
 
Has anyone experienced anything similar?  I couldn't find anything in the forums relating to this.  We installed it in June and have had (2) episodes of it going into standby for no apparent reason, no logs.  Both times we were gone from home for a week. Not sure if that is somewhat related. 


Thank you!

[ClassicCrazy]

I don't have that AIO or those Midnite batteries but have some thoughts.
I once had my system shut down when I first got lithiums and didn't realize that the SOC in them could be very inaccurate. Especially after not filling up all the way after repeated days.
When my system shut down , it also shut down my  Midnite Classic. So no charging could happen. And since my batteries went low they shut down on low voltage .
What I had to do in my case was put some charge in my  batteries to wake up the bms again. The bad thing was that I couldn't  use my Outback inverter to charge from the generator since the batteries were shut off.
Luckily I had an idea to use a 48v ebike charger - very low amps but I used it to charge the batteries from generator. Actually  I only had to use it for less than a minute because as soon as the bms saw the voltage it woke up the batteries and life was good again. It powered up the Classic so it could charge for solar, and I could also charge with the Outback inverter.
At the time I was thinking the bms SOC was correct and the Classic reported SOC ( via it's whizbang shunt ) was wrong. The Classic was more correct .
Fast forward in time to now and I understand how all the lithiums work, how to truly understand the real SOC , and avoid shutdown in the cold short days  of winter.
Since I don't have that AIO I don't know all the settings and how it works. On my Outback there were settings to control how it split power to send some to load and charge with the rest. And I could limit the input AC from the generator too . Since I only have the Honda 2000 it was possible for the generator to shut down on overload but I could limit the AC input so it didn't do that.
I wonder if there is a setting in your AIO that is set too low and that is why  your not charging much from your generator ?
My suggestion for the long run is to consider getting a Victron Smart Shunt which is very accurate about the SOC of the whole battery system once set up properly . I view the SOC from Victron on Home Assistant which I use for all my solar stuff, but it also works with the Victron phone app.
Seems like most people have run into the bms reported SOC being wrong problem on most any lithium batteries .
So in summary don't rely on battery bms reported SOC .
I can tell you how to verify that your batteries are full by watching the voltage at the top charge, and watching the current . When the batteries are full the current will drop.
Does Midnite have monitoring software for your batteries so you can see what is going on in them  - voltage, current, all cell voltages and any balancing ?
Are you using closed loop where the batteries tell the AIO when to charge and not charge ?
That can be an issue if that doesn't work correctly . If you can set the AIO to charge in open loop mode using Lead Acid settings that might be  your best bet for now . I think Midnite had info the battery specs for which settings to use for absorb, absorb time , and float. The battery firmware update might be to change bms setpoints .
Watch my video I made about SOC being way off if you want. Different battery , different bms but the concepts are the same.
https://youtu.be/eLsq2q8eHfQ
Larry

[BlackwaterPark]

At the time I was thinking the bms SOC was correct and the Classic reported SOC ( via it's whizbang shunt ) was wrong. The Classic was more correct .

Larry

That interesting, and begs the question about the protocols the batteries BMS uses to determine whether the battery should take the charge, throttle back, start its passive balancing routine, or go into float. Without knowing at all, I would *hope* that it is using the voltage readings across the entire cell pack to do this, and not the SoC readings.  Rick did tell me on the phone that I would get a better charge running open, so that makes me wonder. Why would that be? Will the hawkes bay take priority over the bms or something when you go with open loop? And if your going open loop, is there any reason to keep the batteries linked to each other via canbus, if they are not ultimately linked to your inverter or CC?

[RightRudder]

I don't have that AIO or those Midnite batteries but have some thoughts.
I once had my system shut down when I first got lithiums and didn't realize that the SOC in them could be very inaccurate. Especially after not filling up all the way after repeated days.
When my system shut down , it also shut down my  Midnite Classic. So no charging could happen. And since my batteries went low they shut down on low voltage .
What I had to do in my case was put some charge in my  batteries to wake up the bms again. The bad thing was that I couldn't  use my Outback inverter to charge from the generator since the batteries were shut off.
Luckily I had an idea to use a 48v ebike charger - very low amps but I used it to charge the batteries from generator. Actually  I only had to use it for less than a minute because as soon as the bms saw the voltage it woke up the batteries and life was good again. It powered up the Classic so it could charge for solar, and I could also charge with the Outback inverter.
At the time I was thinking the bms SOC was correct and the Classic reported SOC ( via it's whizbang shunt ) was wrong. The Classic was more correct .
Fast forward in time to now and I understand how all the lithiums work, how to truly understand the real SOC , and avoid shutdown in the cold short days  of winter.
Since I don't have that AIO I don't know all the settings and how it works. On my Outback there were settings to control how it split power to send some to load and charge with the rest. And I could limit the input AC from the generator too . Since I only have the Honda 2000 it was possible for the generator to shut down on overload but I could limit the AC input so it didn't do that.
I wonder if there is a setting in your AIO that is set too low and that is why  your not charging much from your generator ?
My suggestion for the long run is to consider getting a Victron Smart Shunt which is very accurate about the SOC of the whole battery system once set up properly . I view the SOC from Victron on Home Assistant which I use for all my solar stuff, but it also works with the Victron phone app.
Seems like most people have run into the bms reported SOC being wrong problem on most any lithium batteries .
So in summary don't rely on battery bms reported SOC .
I can tell you how to verify that your batteries are full by watching the voltage at the top charge, and watching the current . When the batteries are full the current will drop.
Does Midnite have monitoring software for your batteries so you can see what is going on in them  - voltage, current, all cell voltages and any balancing ?
Are you using closed loop where the batteries tell the AIO when to charge and not charge ?
That can be an issue if that doesn't work correctly . If you can set the AIO to charge in open loop mode using Lead Acid settings that might be  your best bet for now . I think Midnite had info the battery specs for which settings to use for absorb, absorb time , and float. The battery firmware update might be to change bms setpoints .
Watch my video I made about SOC being way off if you want. Different battery , different bms but the concepts are the same.
https://youtu.be/eLsq2q8eHfQ
Larry

Larry,,,thank you for the response and thoughts from your experience.  I will see if I can answer your questions. 

I was on the phone with Rick from Midnite for half a day trying to trouble shoot the issue.  We adjusted settings in the app to try and trick the batteries, changed charge settings and nothing seemed to work.  The generator just didn't want to up the charge at all.  It seems to me that it is being limited by either the batteries or the inverter because of the low SOC. We even made them open loop I believe by unplugging the linked communication to each battery and that didn't work.  It is normally closed loop communication if I understand that correctly, the batteries are all linked together then linked to the inverter via comms cables. 

I have noticed though, when we do get brief spurts of good solar input, the generator charge goes up as well but not by much.  Unfortunately we are not seeing any sun right now for several days.

I have read about waking up the BMS but I am not sure if mine is on or not currently.  It is my understanding that if I have a sleeping BMS, I won't have any SOC charge lights?  I currently have the last SOC light which is indicating a range of 0-17%.  The green run light on the batteries is also blinking which tells me the batteries BMS is alive?  The app shows my bank at 7%, up from 0% over 2 full days of trying to charge via solar and generator.  In talking with Rick I would assume he would have brought up the BMS. 

I can view via the Midnite app the battery current, voltage, charge power, discharge power, SOC and SOH.  I don't think I have the ability to see individual cells with the app.  The more I research and learn, it seems like a 3rd party monitoring app is needed that gives me all the information you are referring to, such as the Victron Smart Shunt or Solar Assistant.

I am still waiting to hear back from Rick, as he was requesting the firmware updates from the engineers.  I understand the BMS system is to protect the batteries but it is so frustrating to not be able to recover from a failure and charge with my generator.  Then to your point that you can't rely on the data for SOC that comes with the equipment you bought is even more frustrating.

Everyone I have talked to from various vendors have all had different charging parameters for the batteries. Some say live between 20-80% and charge to 100% once in a while.  Now I am being told to charge to 100% 2-3 times a week to keep the cells balanced.  I understand it is on me to learn, but the volume of different information is crazy.  What is your recommendation based on experience for charging?  Even during the summer, when 100% happens a lot via solar only, is it adequate for the batteries?  Or should I be supplementing charging them with a generator to make sure they get a full 100% and the cells are balanced?

Again, thank you so much for your time and knowledge as I continue to learn each day!

Jeff

[RightRudder]

I will keep this post updated in the hopes it creates conversation yet more importantly helps others in their solar adventures.

I just spoke with Ronald at Midnite Power.  We are currently updating the inverter and then will update the batteries once they have sufficient charge.  He did say that the BMS is on as I suspected.  I asked about the generator issue.  He said it could be that the frequency has gotten out of wack over time, especially with older generators.  The new firmware has a wider range for frequency so that may fix it.  He did say that if I put a new generator on it that it would probably work fine. 

Stay tuned!

[ClassicCrazy]

Jeff - I understand your confusion. The truth is it took me a long time to understand all the conflicting bits of Lithium , open loop, bms, etc information and opinions.
I can give you a summary
When in closed loop the bms sends data to the charger ( your aio or in other cases to a charge controller). The charger uses that information to charge or turn off charge . There is a mess of how different manufacturers use what information from the bms . I wish it was plain as day but in many cases it isn't.
Bottom line - if those communications aren't working right for whatever reason , no charge or the charge may be limited.
In Open Loop we just set charge setpoints and ideally the charge controller just uses those to know when to charge , absorb voltage, absorb time, and then go to float. That is it in a nutshell.
Many of the new charge controllers like the Aio or many others on the market like EG4 or Growatt, etc - they are a charge controller and inverter in one box.
Now to the battery bms - in a nut shell the bms protects the cells. It is usually programmed with setpoints that you can't control in  your case. So those setpoints tell the charge controller to charge or quit charging ( and maybe limit current while charging).
In my case , I have batteries with bms that I can get into and change those bms setpoints. So over a few years I have changed and tweaked things . At first I didn't know what was correct .
I watched lots and lots of Off Grid Garage videos on Youtube. He has been leading or influencing a lot of the DIY lithium community and even some bms manufacturers. He takes time to test and explain in detail what he sees and recommends. I find his advice very good and lots of other agree.
So what does this all mean ?
oh wait there is one more concept . Cell balancing. There are two methods that most bms accomplish this by. One is passive where there are shunt resistors bypass extra current of a high cell so the low cells can fully charge up. Otherwise some cells go high and others don't get enough - but when all the cells voltages are added up the overall battery voltage looks good. Keep in mind that this only happens at the very top of the charge. And balancing depends on what the balance setpoint is set to in the bms. In my case I can set this. In your case sounds like the tech you are working with is himself workign with someone to make a firmware update to tweak all these settings.
The passive balance has drawbacks in that it is usually small amounts of power that can be bypassed. Like maybe 250 ma or less. So on a 100ah or 300 ah cell this would take a really long time to balance. If the system is charging at 2 amps that is 2000 ma so you see the issue most systems charge 10 or 20 amps or more and the passive balance can hardly keep up and even if limited current to like 1 amp it will take many hours to balance cells ( that is if they are way out of balance.
The other type of balancing is called Active . Instead of bypassing or shunting power around the high cell, in active balancing it can take 2 amps or more from a high cell and move it to the low cell. It keeps cycling like this from  high cell to low cell no matter which cell is high or low , until it gets all cells balanced. this is much quicker process. There are few bms that I know of that do this - my JKbms does, and on some of my old batteries I installed a separate active balancer that will do this .
But getting back to your question of what SOC % to charge or discharge to.
If the SOC is accurate and the cells are balanced ,then there is no reason to not charge close to 100% and stop taking power out before the bms shuts the battery off on cells going under voltage which in theory would be 0% soc. In practical sense you would stop at at least 10% to avoid that bottom shut off.
But in the real world not everything is perfect sometimes. If the cells aren't balanced that bottom zone of 0% will come earlier since some were never filled up all the way at the top.
What this means is that even with cells unbalanced you can still draw lots of power out of the middle lets say 20% to 80% soc without even having to worry about cell imbalances. But you give up some power capacity .
We like the cells to be balanced . Some of that is difficult due to the cells from factory never being tested and matched well by battery manufacturer when they were selected and put together. The easier part is using active or passive balancing to smooth out the slight imperfections , but remember balancing only happens at the very top of the charge .
I can talk some more about this but gotta do some stuff here.
If we could see all your cells are doing at the very top of the charge cycle we could analyze the situation you have.
Here are the setpoints I  use for LiFePo4 as advised by Off Grid Garage and others.
Absorb to 55.2v ( 3.45v per cell )which is 99% full. We stay away from the last 1% because there is little gain and the charging curve is very steep there so not worth it. And that way you stay away from the critical charge stop point of 3.65v per cell .
I hold that absorb of 55.2v for an hour to give the bms time to do it's thing and balance cells.
After an hour I go to float voltage of 53.6v which is the point where the cells don't charge or discharge. Some inverters like that float to be a little bit higher up to 54v .
And in the real world of my  system where I have some voltage drop in cables that i want to compenstate for I charge a bit higher than 55.2v .
My cell balancing starts to 3.45v and will turn off when the cells are either within 5mv differential of each other , or drop to 3.4v .
From what I see a lot of people ( manufacturers and or techs ) are always learning and evolving their thoughts and theories of what is the best way to charge lithiums.
As I said, after watching Andy on Off Grid Garage do extensive testing on cells to various setpoints and then discharge and see capacity, I use his conclusions which are above.
Even if the manufacturer doesn't supply software to see all the cells in a live time format, there are ways to get that data from the batteries communications. But you might have to have some geek skills with linux computer or raspberry pi to do that.
I have done both ways and get good detailed data that can be graphed as you saw in the video I linked if you watched that.
Larry

[ClassicCrazy]

I will keep this post updated in the hopes it creates conversation yet more importantly helps others in their solar adventures.

I just spoke with Ronald at Midnite Power.  We are currently updating the inverter and then will update the batteries once they have sufficient charge.  He did say that the BMS is on as I suspected.  I asked about the generator issue.  He said it could be that the frequency has gotten out of wack over time, especially with older generators.  The new firmware has a wider range for frequency so that may fix it.  He did say that if I put a new generator on it that it would probably work fine. 

Stay tuned!

Maybe you can ask him what setpoints they are going to program the bms to as I mentioned above.
That would be interesting to know. It would also be interesting to know what brand bms is in your battery even if it is somewhat custom made one.
Take a look here https://off-grid-garage.com/my-settings/  which is what I use for my JK inverter BMS ( and in my Pace BMS)
Larry

[ClassicCrazy]

In closed loop some of the bms reported info that the controller may use is the high and low cell data. So if one cell is showing low ( under voltage protection UVP) , that can prompt it to shut down , or if too high cause it to stop charging ( over voltage protection OVP .
It gets back to the balancing issues at the top and bottom of charge / discharge.
Stay in the middle no problems, but try to get at very bottom to get all the capacity out of battery can shut down in closed loop if a cell goes UVP
Larry

[BlackwaterPark]

Good Evening,

Newbie and DIY learn as I go here, so go easy!

I am totally off grid, have a MN15-12KW-AIO and (4) MN PowerFlo5 batteries.  Our system has been working and just (2) nights ago it went totally dark.  Batteries went from 56% to 0% in the blink of an eye.  I have been in contact with Rick who has been a huge help.  He is working with me on getting firmware updates completed. 

In trying to charge the batteries back up, it has been painfully slow.  We have very little solar input due to the weather, and the generator is only putting in .018KW.  The generator is a Honda EU6500is.  It kicks up when the solar input increases, but when it disappears it drops off again.  I have gone from 0% to 8% in (2) days on the batteries.  The generator worked just fine prior to this issue charging the batteries.  The generator will still run everything else, but it is very limited on the input to the batteries. Does anyone have any thoughts on how to make this go faster?  Rick had helped me try to trick the batteries with changing settings and that didn't work. I am just throwing a bone out there as Rick tried all the tricks he had. 
 
Has anyone experienced anything similar?  I couldn't find anything in the forums relating to this.  We installed it in June and have had (2) episodes of it going into standby for no apparent reason, no logs.  Both times we were gone from home for a week. Not sure if that is somewhat related. 


Thank you!

are you only trying to charge batteries via the charger on the AIO? I realize you have pv, but you aren't really getting much from the sun. For what it's worth, I've found that the chargers on inverters are finicky, especially with the THD ratings...most seem to need it to be less than 3%, which most gensets do not provide (although Hondas typically do), and the Hz coming in. I've had to make adjustments to my idler before to adjust the freq outbound into the charger. After a couple frustrating years of this, I simply went back to using a dedicated discrete ac to dc charger going straight into the battery bus bars. They just work. The EG4 chargeverter is a really good one, but I'd steer clear of the ecoworthy one, as LOTS of folks seem to have major issues with that one.

[RightRudder]

Okay as promised, here is an update on my issue.  I first off want to thank you guys for responding and giving me information based on your knowledge and experience.  I am new to this and have learned a lot over the last year.  I like the challenge, but it can be overwhelming.  So thank you again!!

The issue boiled down to the firmware update.  In the old firmware, the SOC would slowly decalibrate the SOC if you didn't bring the batteries on a regular basis to 100% for the passive balancing.  What is a regular basis?  I was told once in a while, 2-3 times a week, read different forums online and the consensus seemed to be all over the board.  Midnite Power told me today, once a week they should be charged to 100%.  It would be super cool if that was in the battery manual.  So, as I wasn't charging them to 100% on a regular basis, the SOC was out of calibration and the end result ended up in the batteries going to 0% even though the SOC said they were good.  The new firmware is supposed to help manage that better. 

The second issue after this happened, was my generator was not charging the batteries to get me back in service.  It was explained that over time, with generator charging, the frequency can become out of wack, and the inverter only had a small threshold for frequency changes, especially with older generators.  The new firmware has a broader range for frequency changes, thus fixing my generator charging issue.  Everything is working great now and getting charged up!

Again, thank you for the help, especially Larry!!  I truly appreciate it.  Most of the folks on here have a great deal of knowledge to tap into and I am thankful you all are willing to share your knowledge. 

[RightRudder]

I will keep this post updated in the hopes it creates conversation yet more importantly helps others in their solar adventures.

I just spoke with Ronald at Midnite Power.  We are currently updating the inverter and then will update the batteries once they have sufficient charge.  He did say that the BMS is on as I suspected.  I asked about the generator issue.  He said it could be that the frequency has gotten out of wack over time, especially with older generators.  The new firmware has a wider range for frequency so that may fix it.  He did say that if I put a new generator on it that it would probably work fine. 

Stay tuned!

Maybe you can ask him what setpoints they are going to program the bms to as I mentioned above.
That would be interesting to know. It would also be interesting to know what brand bms is in your battery even if it is somewhat custom made one.
Take a look here https://off-grid-garage.com/my-settings/  which is what I use for my JK inverter BMS ( and in my Pace BMS)
Larry

Larry,

I have a list of questions that I want to call them back with next week.  I will certainly add yours to my list and see if they can answer them. 

Jeff

[Brucey]

Okay as promised, here is an update on my issue.  I first off want to thank you guys for responding and giving me information based on your knowledge and experience.  I am new to this and have learned a lot over the last year.  I like the challenge, but it can be overwhelming.  So thank you again!!

The issue boiled down to the firmware update.  In the old firmware, the SOC would slowly decalibrate the SOC if you didn't bring the batteries on a regular basis to 100% for the passive balancing.  What is a regular basis?  I was told once in a while, 2-3 times a week, read different forums online and the consensus seemed to be all over the board.  Midnite Power told me today, once a week they should be charged to 100%.  It would be super cool if that was in the battery manual.  So, as I wasn't charging them to 100% on a regular basis, the SOC was out of calibration and the end result ended up in the batteries going to 0% even though the SOC said they were good.  The new firmware is supposed to help manage that better. 

The second issue after this happened, was my generator was not charging the batteries to get me back in service.  It was explained that over time, with generator charging, the frequency can become out of wack, and the inverter only had a small threshold for frequency changes, especially with older generators.  The new firmware has a broader range for frequency changes, thus fixing my generator charging issue.  Everything is working great now and getting charged up!

Again, thank you for the help, especially Larry!!  I truly appreciate it.  Most of the folks on here have a great deal of knowledge to tap into and I am thankful you all are willing to share your knowledge. 

The easiest way to avoid issues where bms soc% deviates from reality is to use a victron smartshunt to monitor soc, as it can pick up very small current that a battery bms may not "see". Then set inverter to shutdown at say 47V, back on at 48V. A smartshunt can also drift but far less than a battery bms. Then a full charge once a week or two should be sufficient to reset to 100%.

This way you won't be caught by surprise with a sudden shutdown.

[ClassicCrazy]

The Victron Smart Shunts are also easy to get the data into monitoring or home automation programs. It is my point of reference for what is really happening with my batteries.
I always thought the Whizbang is also very accurate , but the nice thing about Victron Smart shunt is it works stand alone.
Larry