Unable to get jupyterlab-git extension working

JupyterHub The Littlest JupyterHub
, ,
1

We are currently in the process of automating AWS EC2 image builds that have TLJH installed along with some basic extensions. I’m using an Ansible playbook to handle the TLJH installation. When the EC2 instance is launched, we have a path specified in our load balancer to redirect to the hub server on there.

I am able to successfully access JupyterLab via our URL and it works fine. However, I am unable to see the JupyterLab-Git panel in the UI. I have even tried manually installing the extension in the EC2 shell; it shows up in the server and lab extension lists but does not show up in the UI even after reloading tljh. Unfortunately, I’ve been down this rabbit hole for two whole days with nothing to show for it, except a ton of frustration.

I really need someone to lay out the exact commands (including venv activation) required to successfully install extensions and have their corresponding UI elements show up in JupyterLab after reloading/respawning the user’s server.

Thanks in advance…

2

First thing to check is what are the outputs of

jupyter server extension list

and

jupyter labextension list

also, worth checkig if the version of jupyterlab-git and jupyterlab are compatible

1 Like
3

Server extension list:

Config dir: /root/.jupyter

Config dir: /opt/tljh/hub/etc/jupyter
    jupyter_lsp enabled
    - Validating jupyter_lsp...
      jupyter_lsp 2.2.5 OK
    jupyter_server_mathjax enabled
    - Validating jupyter_server_mathjax...
      jupyter_server_mathjax  OK
    jupyter_server_terminals enabled
    - Validating jupyter_server_terminals...
      jupyter_server_terminals 0.5.3 OK
    jupyterlab enabled
    - Validating jupyterlab...
      jupyterlab 4.1.6 OK
    jupyterlab_git enabled
    - Validating jupyterlab_git...
      jupyterlab_git 0.50.0 OK
    nbdime enabled
    - Validating nbdime...
      nbdime 4.0.1 OK
    notebook_shim enabled
    - Validating notebook_shim...
      notebook_shim  OK

Config dir: /usr/local/etc/jupyter

Lab extension list:

JupyterLab v4.1.6
/opt/tljh/hub/share/jupyter/labextensions
        jupyterlab_pygments v0.3.0 enabled OK (python, jupyterlab_pygments)
        nbdime-jupyterlab v3.0.1 enabled OK
        @jupyterlab/git v0.50.0 enabled OK (python, jupyterlab-git)
4

The extension needs to be installed in the user environment, not the hub environment.

2 Likes
5

The extension needs to be installed in the user environment, not the hub environment.

Are you referring to activating the /opt/tljh/user venv prior to installation? If so, I have already tried that and it still will not show up when relaunching the server.

FWIW, I did a vanilla install of JupyterLab on an Ubuntu container via pip install jupyterlab, installed the extension via pip install jupyterlab-git and the corresponding UI panel showed up on the first try. This would lead me to believe that there is an underlying issue in trying to get this working successfully on TLJH.

1 Like
6

I’ve just deployed a new Ubuntu 22.04 VM, installed TLJH following

Installed jupyterlab-git from an admin terminal

sudo -E mamba install jupyterlab-git

and it’s working fine for me, for admin and non-admin users:

image
image1280×816 55.7 KB

I’ve also verified that installing with pip works in an admin terminal sudo -E pip install jupyterlab-git

1 Like
7

@manics - Thanks. However, I would like to install extensions outside of the IDE since we’re creating EC2 instances via Packer/Ansible and need TLJH, along with the extensions, installed when building the image.

Please let me know the commands to successfully install extensions at the OS level.

8

The extension needs to be installed in the user environment, not the hub environment.

This turned out to be the key component. I don’t know why the manual install to user venv wasn’t working but through Ansible the extension installed and rendered successfully.

1 Like
9

Unfortunately we’re back to this not working again. This is on an AWS EC2 running Ubuntu 22.04. Here is the output of the extensions list from within the terminal:

jupyter-6c9a7085-c87a-412c-afba4@ip-10-74-41-147:~$ jupyter server extension list
Config dir: /home/jupyter-6c9a7085-c87a-412c-afba4/.jupyter

Config dir: /opt/tljh/user/etc/jupyter
    jupyter_lsp enabled
    - Validating jupyter_lsp...
      jupyter_lsp 2.2.5 OK
    jupyter_resource_usage enabled
    - Validating jupyter_resource_usage...
      jupyter_resource_usage 1.0.2 OK
    jupyter_server_mathjax enabled
    - Validating jupyter_server_mathjax...
      jupyter_server_mathjax  OK
    jupyter_server_terminals enabled
    - Validating jupyter_server_terminals...
      jupyter_server_terminals 0.5.3 OK
    jupyterlab enabled
    - Validating jupyterlab...
      jupyterlab 4.2.0 OK
    jupyterlab_git enabled
    - Validating jupyterlab_git...
      jupyterlab_git 0.50.1 OK
    nbdime enabled
    - Validating nbdime...
      nbdime 4.0.1 OK
    nbgitpuller enabled
    - Validating nbgitpuller...
      nbgitpuller 1.2.1 OK
    notebook enabled
    - Validating notebook...
      notebook 7.2.0 OK
    notebook_shim enabled
    - Validating notebook_shim...
      notebook_shim  OK

Config dir: /usr/local/etc/jupyter

jupyter-6c9a7085-c87a-412c-afba4@ip-10-74-41-147:~$ jupyter labextension list
`sys_prefix` level settings are read-only, using `user` level for migration to `lockedExtensions`
JupyterLab v4.2.0
/opt/tljh/user/share/jupyter/labextensions
        nbdime-jupyterlab v3.0.1 enabled OK
        jupyterlab_pygments v0.3.0 enabled OK (python, jupyterlab_pygments)
        @jupyter-server/resource-usage v1.0.2 enabled OK (python, jupyter-resource-usage)
        @jupyter-widgets/jupyterlab-manager v5.0.10 enabled OK (python, jupyterlab_widgets)
        @jupyter-notebook/lab-extension v7.2.0 enabled OK
        @jupyterlab/git v0.50.1 enabled OK (python, jupyterlab-git)

To me, it looks like the extension is installed and enabled in server and user, but not rendering in the UI. I really do not understand what I am missing.

10

It sounds like you managed to get it working before- what’s changed in your Ansible?

1 Like
11

@manics - Below is the content of the Ansible task file:

---

- name: Generate a 32-byte hex string
  ansible.builtin.set_fact:
    jupyter_token: "{{ lookup('community.general.random_string', length=64, override_all='0123456789abcdef') }}"

# Research Ansible way to download and execute/pipe script    
- name: Run Jupyter setup
  ansible.builtin.shell: "curl -L https://tljh.jupyter.org/bootstrap.py | python3 - --admin golden_ami_admin"
  become: true

- name: Store token in file
  ansible.builtin.copy:
    content: "{{ jupyter_token }}"
    dest: /opt/tljh/config/jtoken

- name: Set Jupyter port
  ansible.builtin.command: "tljh-config set http.port 8000"
  become: true
  notify:
    - TLJH Reload Proxy
    - TLJH Reload Hub
    - TLJH Reload All

- name: Clone jwt authenticator repo
  ansible.builtin.git:
    repo: https://github.com/izihawa/jwtauthenticator_v2.git
    dest: ~/jwtauthenticator_v2
    clone: true
  become: true

- name: Copy jwt authenticator
  ansible.builtin.copy: 
    src: ../files/auth/jwtauthenticator.py
    dest: ~/jwtauthenticator_v2/jwtauthenticator/jwtauthenticator.py
  become: true

- name: Copy jwt setup
  ansible.builtin.copy: 
    src: ../files/auth/setup.py
    dest: ~/jwtauthenticator_v2/setup.py
  become: true

- name: Copy jwt.py config file
  ansible.builtin.template: 
    src: ../templates/jwt.py.j2
    dest: /opt/tljh/config/jupyterhub_config.d/jwt.py
  become: true
  
- name: Install jwt authenticator (hub)
  ansible.builtin.pip:
    name: .
    editable: true
    chdir: ~/jwtauthenticator_v2
    virtualenv: /opt/tljh/hub
  become: true

- name: Copy user-setup script
  ansible.builtin.template: 
    src: ../templates/user-setup.sh.j2
    dest: ~/user-setup.sh
    mode: +x
  become: true
  
- name: Set Jupyter authenticator 
  ansible.builtin.command: "tljh-config set auth.type jwtauthenticator.jwtauthenticator.JSONWebTokenAuthenticator"
  become: true
  notify:
    - TLJH Reload Proxy
    - TLJH Reload Hub
    - TLJH Reload All

- name: Copy requirements.txt
  ansible.builtin.copy: 
    src: ../files/requirements.txt
    dest: ~/requirements.txt
  become: true
  
- name: Install JupyterLab extensions
  ansible.builtin.pip:
    requirements: ~/requirements.txt
    virtualenv: /opt/tljh/user
  become: true
  notify:
    - TLJH Reload Proxy
    - TLJH Reload Hub
    - TLJH Reload All

Notes:

  • The service API token is generated to be used when a user setup script is run upon the EC2 instance’s launch.
  • We’re using a custom JWT authenticator whose configuration is stored in a jwt.py config file.
  • The requirements.txt file is meant to contain extensions to be installed. Right now, it contains jupyterlab and jupyterlab-git.

Please advise on where I should look next.

12

Just to confirm, this occurs reproducibily on every deployment of a new VM?
Can you try:

  • Replacing ansible.builtin.pip with a ansible.builtin.command to run /opt/tljh/user/bin/pip install requirements.txt in a new VM
  • If that doesn’t work then try ansible.builtin.shell to activate the virtualenv and then run pip (something like . /opt/tljh/user/bin/activate && pip install requirements.txt) in a new VM
13

@manics - Yes, this occurs every single time on a new VM. Results of your suggestions:

  • Using ansible.builtin.command did not make a difference. The command itself was successful but the extension did not render in the UI. Running the same command in the VM’s shell and reloading TLJH did not resolve either.
  • Using ansible.builtin.shell did not work either. Similarly, running the same command in the VM’s shell and reloading TLJH did not render the extension.
14

FWIW I have even tried manually activating the user env and running pip install and it still doesn’t show up in the UI. Another thing I’ll note is that we are using nginx on the EC2 with a reverse proxy to forward requests from the AWS load balancer to the instance.

I really want to avoid having to install the full-blown JupyterHub and Lab since it seems like an incredibly difficult and convoluted task. Any other ideas would be greatly appreciated.

15

Unfortunately I don’t think there’s an easy answer, especially since you managed to get it working but then it stopped working again.

If it helps the user environment is decoupled from the hub environment, so the only difference between running jupyterlab from a terminal as a standard user, and launching jupyterlab using JupyterHub, will be the spawner:

Which is based on GitHub - jupyterhub/systemdspawner: Spawn JupyterHub single-user notebook servers with systemd
so it’s probably worth comparing that behaviour to rule the spawner in/out.

After that you’ll probably need to rule out problems with Ansible tasks (either your definition of them, or a bug in the Ansible module). For example, if you remove the user environment tasks from Ansible and manually create the environment does it work (test with the spawner, and from the command line without JupyterHub)? If you don’t use Ansible, but manually run the equivalent shell commands for each task, does it work (presumably it does!)?

1 Like
16

@manics - I would appreciate clarity on the following:

The User Environment is a conda environment that is shared by all users in the JupyterHub. Libraries installed in this environment are immediately available to all users. Admin users can install packages in this environment with sudo -E

  • What is the exact shell equivalent of installing packages / extensions via an admin user terminal? Which virtual environment should be activated prior to installing?
  • Does it make a difference if pip or conda is used to install these packages? If yes, then why?
  • After installing extensions what are the commands that need to be executed to ensure the packages are properly included in the server and UI? I have seen references to using either tljh-config reload proxy/hub and restarting the systemd services. Does one of the virtual environments need to be active prior to reloading?
17

/opt/tljh/user/bin/pip install jupyterlab-git run as root works for me (in a console shell, it’s not mandatory to use a JupyterHub admin terminal). You can activate the virtualenv (. /opt/tljh/user/bin/activate) if you want, but using the full path to /opt/tljh/user/bin/pip works for me.

Conda packages usually take the original source, and repackage it as a conda package. It’s usually behaves the same as installing from pip, but that’s not mandatory, it’s up to the Conda packager.

You shouldn’t need to restart JupyterHub- the hub doesn’t care how your JupyterLab is configured. If you’re already running a singleuser server (JupyterLab) you’ll obviously need to stop and start that though.

18

After stepping away from this for a while, I just looked at this issue again with a fresh set of eyes and was able to figure out the source of the problem.

Part of our Ansible playbook applies nginx hardening (we use nginx as a reverse proxy on the EC2 instance to redirect to the running Jupyter server). Among the hardening settings that get applied, one of them is for limiting connections (limit_conn). Commenting out this setting resulted in the extension’s GUI being rendered as expected.

1 Like