It may not be strictly possible to do this within configuration.
If streamlit doesn’t support dynamically dispatching to multiple apps on the same port, you’ll probably have to implement your own mechanism.
Still in jupyter-server-proxy, you could write a simple “server” that just starts another server, per file, on a new port, and serves a one-line redirect HTML to {base_url}/proxy/{new_port}. No idea how you’d clean them up, though!
The weirdest thing might be to run streamlit inside the jupyter_server: it looks like streamlit uses tornado under the hood: you could probably figure out some way to use the “guts” of it more directly, and implement multiple app endpoints yourself, as a serverextension, where they share the same app… but this is kinda crazy.