From what I can tell, the only way to let the system know something's gone wrong is to raise a RuntimeError() exception. This kind of works, but it also makes a mess, filling JupyterLab's error output (stderr) with tons of scary and confusing text. I'd really prefer just a couple lines of clean log.error() messages instead in stdeout with copy of them to JupyterLab's frontend user is looking at.

Raising exceptions is the way startup errors are expected to be propagated. How those exceptions are displayed to the user is a different matter (and probably something to deal with in the Lab layer).

But sometimes, especially when the kernel needs to run on another computer (a remote host), this might not be possible due to some infra or scheduling issues. This means the kernel can't start, and there's no connection info will be returned from the call.

Starting remote kernels introduces multiple layers in which errors (and delays) can occur. Work was done in Jupyter Server on Pending Kernels that you may want to look at. IIRC, they need to be enabled, but provide some better handling for startup delays specifically for these kinds of issues. However, if your failure is a hard failure (and not just due to things taking longer) raising an exception is what is expected. (This was true even prior to provisioners, albeit the "provisioner" was essentially the Popen call.)

You might also take a look at the Gateway Provisioner classes. It provides a RemoteProvisionerBase class, as well as a ContainerProvisionerBase (if you're working with containers), that can be subclassed while providing most of the necessary infrastructure. If you have questions about these, please open an issue in that repo for further discussion.