Widespread issues in python-roborock in 3.10.x

[allenporter]

We're seeing widespread issues introduced resulting in timeouts, rate limiting. We'll use this issue to track the diagnosis and improvements.

We can also use this to discuss local testing with new library versions inside of Home Assistant for users who are comfortable with making local python package updates.

[Lash-L]

#638 #640

[Lash-L]

We should also look through #632 and double check all the logic there.

[Lash-L]

@rccoleman reports no timeouts or connection loss messages after running for the last 3 hours on 3.10.3

[allenporter]

@rccoleman appreciate the testing.

[allenporter]

In summary we believe the two issues are and solutions are:

• Local connections were skipped entirely, causing large load on MQTT servers. This is the opposite intended consequence of the overhaul with a bug intended to reduce duplicate code that introduced it without sufficient tests. We now have tests in place for this behavior.
• Local connections were being skipped on the first RPC. The check for "can i use the local connection?" would skip since it ran before we actually attempted to create the local connection. This was introduced by a chance that tried to speed up startup time by creating devices without blocking on establishing a connection. Now we block on establishing a connection for X seconds, but don't fail if it fails or times out. This will slow down startup time, but avoids a cloud dependency for the first RPC.

[Lash-L]

I am also running 3.10.4 on a local instance. A potential quirk is how it handles the 3am timeout. I will report back on that specifically tomorrow

[allenporter]

My original thinking is (1) local connection should break, then get re-established by the background thread. mqtt connection should be watched by health manager (2) mqtt connection should be alive thus we can say the device is "available" still. Howevr, commands during this period will either fail or run over MQTT.

However, one caveat to that is given the topic keepalive we may not consider the connection healthy. I may want to change the MQTT channel health to be based on the mqtt session instead of the channel (which needs a topic)

However, if mqtt connection is not alive during this time period then it will still be considered not available. in that case, availability should be determined by RPCs failing.

[Lash-L]

Slightly related #643

Will not help with reliability but should help with all entities being available.

[Lash-L]

Tested with the latest version of the library overnight. Behavior looks expected

Will check logs in a bit

[Lash-L]

The only error/warning log I get is this. So users will still get a warning which we might want to change. But that is on HA side and not library. All in all looking good

[Lash-L]

#645 and #647 relate to fixing issues with A01 devices. Only semi-related