Dangerous bug. DShot randomly sending erroneous signal to ESC after disarming

[Jetrell]

I'd experienced a problem when testing. It would cause the copter to burst into life within a short time after disarming.
And in the worse case, it can harm the user when unplugging the battery. As happen to me. And I have the wound to prove it!

In my case, I assume it is some how related to nav_wp_load_on_boot. When it continues to show MISSION LOADED after the landing detector disarms, then I manually disarm the switch. Then push forward on the elevator stick to remove the OSD stats, to show that message on the system messages.
But it is random. Even though the Mission Loaded messages appear every time. The times it bursts into life is maybe one time in 5. And when it does. The motors can start and stop several times. Reaching full throttle and flipping the copter.
Dshot beeper may also be playing a part in this too.

I had noticed it a few times when running 9.0 development testing. Then I loaded 8.01 and 8.0. With both showing it to be there as well.

This reminded me of an issue a guy on Discord showed me a week back... His planes FC rebooted in the middle of the flight a couple of times randomly.
When we looked over the DVR. It was the same issue. But he had both Continuous servo auto trim and the Stats flight Odometer both enabled.
And the OSD was continually showing Saving Settings after he accidentally disarmed in flight, then rearmed.
And as the case with what I've experienced. The saving of the data to the eeprom doesn't timeout.. In his case it kept trying to save until the FC eventually rebooted and crashed his plane over 3km away.

• FC Board name and vendor: F722
• INAV version string: 8.0 - 8.01 - 9.0 development

[breadoven]

I'd have to double check but I don't think it's continuously trying to save settings I think that message gets stuck on the OSD for some reason. As for the rest of the issue ... there's a lot of interaction happening regarding arming/disarming, saving settings, emergency rearming etc so something likely isn't getting reset properly.

[sensei-hacker]

@Jetrell can we get your diff, and do you by chance have anything connected via MSP? Other than Configurator.

[Jetrell]

Nothing via MSP on the copter that bit me.
But from what I have gathered with limited testing since the time of this report. I believe the motors starting is linked to Dshot beeper.
Which would make some sense. Because it does give permission to pulse the ESC's when the FC is disarmed.

The only way I could be sure, is if the FC could log after disarming.

I'd often thought about the usefulness of a mode that could be set to manually stop log recording and override the disarm logging stop.
Because we've had many issues reports over the last years, that occurred after disarming. Similar to this issue.

[breadoven]

I've thought the same regarding logging only being available when armed. Quite often things happen immediately after disarming which the logging misses. It should at least be possible to set the BLACKBOX mode so you can control logging regardless of arm state. Otherwise have logging continue for a set period (via setting) after disarm. I'm assuming this is possible just through simple logic changes and the disarm limitation isn't to do with something more complicated related to how the blackbox works.

One thing I have noticed recently, possibly when disarm is triggered by landing detection, is that sometimes the OSD locks up for several seconds right at the point of disarm. As with this issue it's random. Not sure I remember it happening with analogue OSD so it might be Walksnail related.

[sensei-hacker]

Jetrell writes:

I'd often thought about the usefulness of a mode that could be set to manually stop log recording and override the disarm logging stop.

Breadoven agrees:

I've thought the same regarding logging only being available when armed. Quite often things happen immediately after disarming which the logging misses. It should at least be possible to set the BLACKBOX mode so you can control logging regardless of arm state.

Yeah we do have the BLACKBOX mode which does control logging, if the mode exists, despite being armed. But only in the negative direction. If someone sets up the blackbox mode and flips their blackbox switch on - they probably want blackbox data at that point, don't they.

It seems quite reasonable to me that if someone takes the trouble to set up BLACKBOX mode on a switch, we should perhaps honor their request to turn blackbox off and on.

[Jetrell]

When I was testing #10955. A motor spin-up after disarming occurred once.. And I haven't been able to get it to do it again.

But at the time, I only had logging continuation after disarm set to 10secs.. And it occurred around the 30sec mark.
The interesting things that I'd noticed when it bit me, in the original report, and this time. Is that it's motor 4 that is the most erratic. And this time, motor 4 wouldn't shut down at all. Even after I tried rearming and moving the sticks. So I ended up having to disconnect the battery... It seemed like the FC was locked up.
Just to note. I had the Stats enabled in these tests. Something I never use, but thought it might help to force a failure while testing. Which is something others may experience.

If I can get it to happen again, I'll report more. But if the FC does lockup. Logs likely won't help much.

edit: I re-enabled the Dshot beeper while running these tests as well.. Nothing like having to wear leather gloves when you disconnect the battery. 🤨

[sensei-hacker]

Is that it's motor 4 that is the most erratic. And this time, motor 4 wouldn't shut down at all.

Have you also seen it on other motors?

[Jetrell]

I was doing some more 'props removed' ground testing of this issue. And re-enabled Dshot beeper to do so.

I'd set blasckbox_arm_control to 60 to try and catch it. The interesting part in nearly every case was it occurred a soon as the log extension timer expired.
But i did catch it once, about 24 seconds after disarming.
To view it in the log. Motor 4 operation can only be seen by Amps and eRPM.
Sometimes the motor only pulsed on and off for a few seconds. While other times it can't be shut down, no matter what I did, as can be seen in the log after that point... I had to pull the power.

inav_0011.zip

The issue is tied back to a problem that was in BLheli32, 32.9 and earlier.
This shows the fix in 32.10. Which the ESC in this copter does not have. Its running 32.9. But the BLHeli_32 server is down as is known. So updates aren't available anymore.

Added protection against unintentional spoolups
– Input signal is qualified before starting, and during the first 5s of running,
signal timeout is only 30ms (vs generally 320ms)
– Input signal loss does not lead to new detection of signal type
(unless input line is held high more than 2 seconds so bootloader is entered)
– So when changing input signal type, the ESC must be power cycled
for it to take effect

This was another message I also found from their developers, for the 32.10 fix.

We did what we could to reduce the risk of hazardous events due to invalid or erroneous input signal in 32.10, like not allowing redetection of signal after the initial powerup, shortening timeouts etc. And as far as I know, BF have also addressed their issue with undefined or erroneous signals going to the ESCs.

So in conclusion. It is related to the ESC firmware.. But as can be seen from what sskaug stated. BF has also put in a fix to prevent erroneous signals.
But to add to this. I've been using this as one of my test copters since late 2022. And have been running the same 32.9 fw and the same general INAV settings. Including Dshot beeper and Turtle mode. And up until recently, I'd never experienced this problem.
But of late, it's occurring after disarm with all 9.0 master builds.

Meaning that the same thing could likely occur on less known ESC firmware's that don't have checks in place. And cause even greater injury to someone else.
In any case, it isn't good for the FC to be spitting out undefined PWM data after its disarmed. Which INAV must be doing of late.

[Jetrell]

After more copter testing with current firmware. I have narrowed this issue down to being related to the Dshot protocol.

The occurrence of INAV sending out an uncommanded RC signal to a 4 in 1 ESC after disarming; is still random while using Dshot.
However I have found by using OneShot or MultiShot. The issue has not occurred in countless flights (arm/disarm cycles).

It would appear something else, among other timing issues in Dshot operation, is broken.
So I thought I'd mention it as a workaround. In case someone else gets attacked by their multicopter when disconnecting the battery.

[sensei-hacker]

I think if blackbox_arm_control is set to -1, you can have blackbox logs even when disarmed, controlled by the blackbox mode.

I say "I think" because that's what is documented, but I've never done it on purpose.

[Jetrell]

That's what the ground test log here is from. #10913 (comment)

[sensei-hacker]

Ah yes, thanks Jetrell. Scrolling through this again I had missed that comment.

So I'm understanding what INAV can potentially do is actively drive the outputs low after disarm. Holding it high would typically be preferable for immunity to ground bounce, but that causes Blheli to enter the bootloader.

[Jetrell]

Betaflight addressed the "motor 1 startup" issue on BLHeli_32 32.9 by ensuring a constant, valid signal is sent to the ESC at all times, even when disarmed.
The root cause was that earlier versions of their firmware would sometimes leave the signal line in an undefined or floating state upon disarming.
BLHeli_32 32.9's safety logic misinterpreted these gaps as valid throttle signals, leading to full-throttle "death rolls" or unintended spin-ups.

Betaflight implemented a fix that forces the flight controller to send DShot motor 0 packets immediately and continuously after disarm to prevent the ESC from entering an "uncontrolled state."

I think these are the BF PR's which they made the changes.

#12544 - Send DShot 0 packets on disarm. This is the primary PR that ensures the DShot signal is never left floating.

#12560 - Safety improvement for ESC initialization. This further refined how the FC handles the transition during the arming and disarming phases.

There was also this report in INAV #10606