feat: RayConnector pubsub support #121
Description
Summary
Currently the RayConnector only supports "pipeline" mode connections, i.e. work queue style one-in-one-out communication. Brokerless pubsub functionality should be possible in Ray using just a set of Ray Queues in a shared object such as a dictionary within a Ray Actor. This is not expected to be a highly scalable or performant approach but should give high reliability in PoC scenarios where commitment to message broker infrastructure is not yet desirable and adds friction to the dev exp.
Example
No response
Alternatives
The current alternative for small scale Ray runs would be to use the ZMQConnector (once #116 is merged). Although there are some known issues:
- Pubsub reliability gotchas such as subscribers not connected before first messages published (some of this should be resolved with shared process run state to orchestrate commencement of data exchange).
- Known limitations of current ZMQ Connector use in multihost context (should be feasible to do but requires changes to how zmq address and host ips are used).