The add_analyzer fails to clean up...sometimes

[stonier]

When add_analyzer leaves, it utilises the bond mechanism to shutdown the bond, triggering an unloading of the analyzers on the aggregator side.

Looks like the bondpy mechanism however, isn't reliably ensuring the aggregator gets triggered. This results in an error message when you reload the same analyzers that were not unloaded:

[ERROR] [WallTime: 1472626077.441982] add_analyzers did not add any analyzers to diagnostic aggregator: Requested load from namespace /diagnostics/navi_common_diagnostic_analyzers which is already in use

Difficult to reproduce with small tests. Right now I'm only getting it on a robot with alot of software running. Even there, it does not occur 100%.

[stonier]

Investigating the bond mechanisms in bondpy.

[stonier]

Issue in bond_core#14

[stonier]

Investigating what nodelets do...they have the same problems even though it's cpp. However the loaders there make a service call to the nodelet manager just before breaking the bond. i.e. they are not relying on the 1) the bond informing the other end or 2) the bond breaking to make sure things work.

• unloadNodelet just before bond_break@shutdown
• service call \in unloadNodelet

Could do the same here.

[trainman419]

We've encountered some similar issues when using the add_analyzers service, and the most common case that I've seen is the bondcpp subscriber dropping messages and prematurely dropping the entire add_analyzers node.

All of the add_analyzers nodes are connecting to the same subscriber, and if they send all of their messages at the same time, it can temporarily overwhelm the bond's subscriber queue. the subscriber queue size is 30, and I've seen this happen with about a dozen add_analyzers nodes running (it looks like the bond publishers are sending at 2Hz, but the aggregator queue is only serviced at 1Hz).

I'm working on a patch that moves each bond to its own topic to alleviate this.