Return `plan_result` to the client

[DominicOram]

In bluesky I can run the following:

def my_plan():
    yield from ...    
    return 4

RE = RunEngine()
assert RE.plan_result == 4

However, if I run the same plan through BlueAPI there is no way to get the result.

This would really aide in moving small incremental jobs from GDA to Bluesky.

Acceptance Criteria

• I can get the plan result from the client
• The format for this in GDA is clean e.g. something like plan_result = run_plan(...)

[tpoliaw]

Looking into how this works in bluesky, it looks like plan_result is available on a returned RunEngineResult rather than on the run engine and only then if the run_engine is configured to return a result.

RE = RunEngine(call_returns_result=True)
res = RE(my_plan())
assert res.plan_result == 4

We are not currently setting call_returns_result but as far as I can see, we're not using the returned plan UUIDs either so it shouldn't break anything to set it now.

[tpoliaw]

Not sure how to handle the case where a plan returns something that isn't JSON serializable. Is returning None an option? An error message (that risks being a confusion if a plan returns a string)? Wrap everything in new wrapper types?

class TaskResult(BaseModel):
    result: Any

class UnserializableResult(BaseModel):
    result_type: String

class TaskStatus(BlueapiBaseModel): # existing type
    ... # previous fields
    result: TaskResult | UnserializableResult | None = None

[tpoliaw]

Will add a proof-of-concept PR that returns results in the task_status of the WorkerEvent that is sent when the plan completes and crashes the server if it's not serializable.

[tpoliaw]

This may all be changing (soon?) if we move away from using the message bus (see #1326). In that scenario, we probably wouldn't have WorkerEvents and the final event sent over the stream could be the plan result.

[tpoliaw]

The problem with having the TaskResult/UnserializableResult types is that we don't know ahead of time that a type isn't serializable. The serialization is done in another library (albeit a DLS one (bluesky-stomp)) with some customisation so we can't realistically try serialising and see what happens.

We could potentially limit return types to pydantic models + python builtins? Anything that pydantic can create a TypeAdapter from?

[DominicOram]

Is returning None an option?

I think this is fine.

We could potentially limit return types to pydantic models + python builtins? Anything that pydantic can create a TypeAdapter from?

Yh, this is fine. I would even be tempted to limit this to just builtins. I really envisaged this "feature" as a bit of a stop gap, to get some basic data out of bluesky. If you're trying to return complex types then you should be using other mechanisms (e.g. the event stream)