I don’t understand what you mean by “add in support on the REST API to utilize the configured authenticators to authenticate.”
Authentication happens on the front-end, e.g. with Keycloak a user is redirected from the JupyterHub UI to the Keycloak UI. Assuming they login successfully the user is then redirected back to JupyterHub, and eventually JupyterHub creates an API token that JupyterLab (or any other frontend) uses.
If you’re doing everything with the JupyterHub backend only, then there’s no way to redirect the user to Keycloak.
If you’re writing a custom UI you’ll need to implement authentication with Keycloak there, and give your front-end a privileged token that allows it to create tokens for users.