Intermittent problems rendering saved widget states

I’m looking for trouble shooting advice. I have a small installation of TLJH that I use for a class. We are using some ipywidget based tools, with “Save Widget State Automatically” set. I have set saving as the default for the whole hub.
Current configuration:

jupyter                           1.1.1
jupyter_client                    8.3.0
jupyter-console                   6.6.3
jupyter-contrib-core              0.4.2
jupyter-contrib-nbextensions      0.7.0
jupyter_core                      5.3.1
jupyter_datainputtable            0.8.0
jupyter-events                    0.11.0
jupyter-highlight-selected-word   0.2.0
jupyter-instructortools           0.8.0
jupyter-jsmol                     2022.1.0
jupyter-lsp                       2.2.5
jupyter_nbextensions_configurator 0.6.4
jupyter_Pandas_GUI                0.9.3
jupyter-resource-usage            1.1.0
jupyter_server                    2.15.0
jupyter_server_fileid             0.9.0
jupyter_server_terminals          0.5.3
jupyter_server_ydoc               0.8.0
jupyter-telemetry                 0.1.0
jupyter-ydoc                      0.2.4
jupyterhub                        5.2.1
jupyterlab                        4.3.4
jupyterlab_pygments               0.3.0
jupyterlab_server                 2.27.3
jupyterlab_widgets                3.0.13
jupyterpidaq_menu                 0.1.0

anywidget                         0.9.13
ipywidgets                        8.1.5
jupyterlab_widgets                3.0.13
widgetsnbextension                4.0.13
plotly                            6.00
  • other tools, but these are the widget related ones.

Both my students and I are seeing intermittent cases where a notebook is saved, but when reopened we get the Error displaying widget: model not found error for some of the widgets. Some widgets that do not display are as simple as a textbox and some are complex widgets containing menus and even plotly plots.

I cannot seem to find any commonality among the cases where this is happening. I have two notebooks that I ran one after the other yesterday. One has this problem and the other does not. I am assuming the issue is with the way the server is saving the notebooks and an error is occurring. I have tried copying the notebooks to another instance of jupyter lab, but it appears the problem is with the saved widget state as things are no better, for the notebooks with problems.

I do not think I yet have enough information to create a useful bug report. Any suggestions on things to investigate or document that could help trace this problem down?