Aggressive TEDAPI usage can cause TACO errors/failures

[brontide]

Backstory:

Got the system installed in April, started monitoring in May with PowerwallDashboard. System is 2x PW + 2x Expansion.

Over the last 3 months I have been fighting with outages and failures in various parts of the system. I installed ethernet into the powerwalls which reduced the issues for a while but they kept coming back. Rebooting the powerwalls seemed to resolve it for a while.

Over the last 3 months it's been crazy, I've gotten virtually every error code you can think of ( see attached screenshot ). Support was unable to assist, offering to "update firmware" with no change in behavior.

On a lark I shutdown pypowerwall and the system reverted to normal operation within an hour and has been operating normally for 12 ( second screenshot ). Monitoring the system can break the system < insert Anakin meme >.

I only restarted pypowerwall after making the following changes.

1. Updated the default interval in the agent to 45 seconds and the timeout for both http sections to 10s
2. Increased the pypowerwall caching/graceful degradation PW_GRACEFUL_DEGRADATION=yes PW_POOL_MAXSIZE=3 PW_CACHE_EXPIRE=30 PW_BROWSER_CACHE=60 PW_CACHE_TTL=60
3. I have set my network proxy to 192.168.92.1 to shutdown for 5 minutes overnight and reboot to give the system time to reset.

It would seem that the TACO is extremely under powered and can be overwhelmed with API calls to the point of throwing error codes and taking whole units offline. I suggest that the default polling intervals be increased and the graceful degradation be enabled by default to reduce the strain on the system.

[jasonacox]

Thanks for this @brontide ! Fascinating discovery.

Despite all my efforts, I have never been able to overwhelm my Powerwall 2/+ system with aggressive API calls. Eventually the gateway of the Powerwall+ system will throttle you with "too many requests" (e.g. http 429), throttle (503) or "forbidden" (403) type responses. In fact, those are the responses that pypowerwall looks for to activate cool-down (no request) mode. I suspect the Powerwall 3 design treats this differently. It seems like a design flaw to allow any access point traffic to, and this is only my speculation, degrade the integrity of the CAN bus and impact performance of the system. I do wish to know if this is a common issue that all Powerwall 3 owners are seeing.

Your recommendations seems reasonable, but it does mean that live status of the system will be up to 30s delayed instead of 5s. That may be required if this is in fact more common experience.

PS - I confess, I had to look up TACO (link) - Tesla Asset Controller. I suspect that is the new name of the site/gateway controller.

[brontide]

Given 24 hours to think about it this is may be limited to sites with complicated setups. Each site has one or more PW3/TACO and one is identified as the leader. The leader unit must have all the DC Expansion units ( as far as I can tell CAN bus internally ), the Gateway 3/Backup switch connected via hardwired CAN bus, the remote monitor connects via wireless CAN bus ( really crappy setup IMHO ), and the additional PW3 units are connected only via Ethernet to the leader. The old site controller is gone.

The Ethernet must be acting as a CAN bridge and, as far as I can tell, the pass-through on the leader is just a dumb switch which means collisions could affect real-time performance.

Half of me thinks I could be chasing ghosts. I will continue to see if anyone else is affected and update if I find other sites.

[CosmicProphet]

This may or may not be a useful data point... And I'm new in this codebase.

background: I just got 2x PW3 + 2x Gateways installed at my home a few weeks ago. I needed 2 sets because my home has a split 400A service and this configuration was needed. The PW3 are NOT interconnected to each other, they are separate sites since Tesla doesn't seem to allow load/power balancing or inter-communication in this configuration sigh.

I noticed that the default polling rate was causing constant errors on the the API calls and definitely seemed too aggressive. I was getting 403s, others (and I think timeouts, I dumped my logs so can't look). Backing off to 20s seemed to allow the system to stabilize after a while. I didn't run the test long enough to see if it soaked the PW3 causing other problems. The behavior of the API calls was very non-deterministic with different errors and different response timing and timeouts. That fact that the endpoint responses felt so uncontrolled was worrisome. I was pretty convinced I was soaking the PWs and it could cause other problems. This was consistent behavior on both Powerwall's.

[jasonacox]

the remote monitor connects via wireless CAN bus ( really crappy setup IMHO )

Yikes!! I agree. It feels like they continue to value engineer the Powerwall setup with each incremental version. 😞 The CAN bus is a simple hardwired solution that would not be difficult to install. 100% would void warranty, but I think I would try to hardware it. 😁

@CosmicProphet you may want to try what @brontide discovered:

pypowerwall.env

PW_GRACEFUL_DEGRADATION=yes
PW_POOL_MAXSIZE=3
PW_CACHE_EXPIRE=30
PW_BROWSER_CACHE=60
PW_CACHE_TTL=60

[brontide]

Yes, I have the molex plug and outdoor rated cat6 ( with drain wire ) for hardwiring the remote monitor but didn't want to open that can of worms while the system was being cranky. With a little grunt work I hope that it will fit the same chaseway as the power.

I was wrong, the remote monitor is wireless rs485 which is similar. It's still maddening because they run communications wire to the PW3 and the gateway HAS rs485 terminals... they have just chosen to use wifi "for simplicity". Had they just run 6 wire rather than 4 they could have used 1 cable.

If the system includes Backup Switch or Gateway 3, do not attempt to connect the RS-485 harness to the Backup Switch or Gateway 3 communication connector. It must be connected to Powerwall 3 (or Tesla Remote Energy Meter can be connected to the Site Controller wirelessly as noted below).

One of the primary functions of the Gateway 3 is current monitoring... but they couldn't add another input or permit it to read the rs485 from the monitor and relay that to the PW3!

Note, the main change that really reduced the load is in the telegraph.conf, going into the agent sections and increasing the interval to a much higher value. This reduces the amount of polling that is attempted.

[jasonacox]

Note, the main change that really reduced the load is in the telegraph.conf, going into the agent sections and increasing the interval to a much higher value. This reduces the amount of polling that is attempted.

That's correct, but also, as a note, the way it works is like this:
telegraf -> pypowerwall proxy -> Powerwall

The agent frequency is how often it calls the proxy. If proxy cache TTL is set high enough, it should cause pypowerwall to serve up the cached answer rather than fetch it from Powerwall directly.