JupyterLSP - Handling Magics

JupyterLab extensions
1

Adding support for custom language LSP server.

I am adding support for the jsoniq programming language using the JupyterLSP extension. I was able to get the server to run and respond to requests, however I am running into an issue caused by the usage of magic cells. JSONiq code is written in .ipynb files, cells executing JSONiq code having the magic %%jsoniq written at the head of the cell. This is done as an extension is responsible to detect these cells and execute them within the RumbleDB runtime.

The problem I am facing is that, in the web browser console, the following warning is given for the magic cells: Ignoring inspections silenced for this document and the diagnostics object is displayed after. How could these inspections not be silenced?

The documentation does not seem to address some of these logs

I could not find any details on this kind of warning in either the docs or GitHub issues. I would appreciate some support on how this behavior could be mitigated, that is, displaying diagnostics for magic cells as well.

The language server spec I am using is the following:

"jsoniq_language_server": {
            "version": 2,
            "argv": ["jsoniq_lsp_start"],
            # Adding python is needed due to JSONiq running in python cells with magic.
            "languages": ["jsoniq", "python"],
            "mime_types": [
                "application/jsoniq",
                "text/jsoniq",
                "text/x-jsoniq",
                "text/python",
                "text/x-python",
            ],
            "display_name": "JSONiq Language Server",
            "requires_documents_on_disk": False,
        }
2

At present, to add Jupyter-specific content not supported by the language server, one needs to either:

  • extend the languagee server to understand the non-standard syntax
    • might be hard, and/or unlikely to be reviewed by upstreams
  • extend the TypeScript side of the house with a new labextension, see:

For truly simple cases, some work has started looking into declarative, regular expression-based technique which would live in jupyterlab settings:

This wouldn’t cover all the cases (e.g. complex flags or other configuration to an IPython magic), but enough, usually, to correctly handle syntax highlighting.

2 Likes
3

Thank you so much @bollwyvl for replying this quickly!
You mentioned syntax highlighting, although I was mostly interested in diagnostics. I think the links you provided give an answer to both interests actually.
If my understanding is correct, we would have to create an extension providing an extractor for the jsoniq magic keyword, however what I don’t understand is how that is picked up by the jupyter-lsp. Assuming such an extension exists, would the extractors provide the ‘.jq’ files to the jupyter-lsp which would then display server answers for various queries (for example, diagnostics)?
Also, you mentioned extending the language server to understand non-standard syntax, you meant the jupyter-lsp extension? We have our own language server and thus making additions to it is no issue. Our problem stemmed from the jupyter-lsp extension logging diagnostics but not displaying them.
Thank you so much once again for your time and patience!

4

how that is picked up by the jupyter-lsp

The extension defines an extractor and registers it with the ILSPCodeExtractorsManager.register

The virtual document uses the extractor patterns to find chunks of code that should be handled by another server, and creates another, new virtual document.

non-standard syntax

Without an example, I’m assuming “magic” means, specifically, “a magic for IPython,” which is not a syntax which the “host” language server (e.g. pylsp) understands.

own language server and thus making additions to it is no issue

In this case, a change would be needed to pylsp (or whichever LS is being used for python)… and the results might not be as good.

While a language server (or plugin) could hard-code some of these rules, they are dynamically extensible by user code and third-party kernel libraries.

The "real" solution, of course...

would be to provide a mechanism for kernels to precisely define what special features they support, which that PR starts, by digging into what JSON would be needed.

In this case, starting the kernel would emit a rich output display (or comm) message of the type application/vnd.jupyter-lsp.extractor.v0+json containing all of its special sytaxes, e.g. %%timeit, %pdb, %html which would get picked up by the client. There would probably be a bit of a lag, and some flip-floppy highlighting/diagnostics, as a language server may already be running, vs a new kernel which has just started.

Then, running something like…

import jsoniq_magic

… would emit another display/comm.

5

Than you once again for your replies! Apologies for the confusion related to the language server we are using. We built JSONiqLSP which should only handles jsoniq files, not python ones. However, for jupyter notebooks, we still rely on python notebooks, which where this confusion arises from.

For example, a typical jsoniq cell in jupyter, with the magic, looks like:

%%jsoniq

let $stores :=
[
  { "store number" : 1, "state" : "MA" },
]
...

It seems that indeed what we want is to implement an extractor for these magic headed cells, and the links you have provided are sufficient to figure that one. Thank you once again for the support!