Python-shell and child_process module not found errors with jlpm

JupyterLab extensions
,
1

Hi all, running into some quite frustrating module issues when trying to build a jupyter lab extension I’m working on. Whenever I try to use the python-shell module to run a .py file from my index.ts file, I get error messages like the ones below after running jlpm run build, even though I have the child_process package installed and it’s in the node_modules folder. It seems to be looking for the package in clearly non-existent directories like node_modules/node_modules etc.

GitHub repo here. The main file in question is src/index.ts. I’m a relative newbie so my problem may be something very basic. Can I even use python-shell the way I’m trying to?

Windows 10
Node v14.17.0
npm 6.14.13
jlpm 1.21.1
jupyter lab 3.0.14

Error message:

[
  {
    loc: 'resolving fallback for shared module child_process',
    message: "Module not found: Error: Can't resolve 'child_process' in '/home/mahler7/green-ext/greencode-ext3/node_modules/python-shell'",
    details: "resolve 'child_process' in '/home/mahler7/green-ext/greencode-ext3/node_modules/python-shell'\n" +
      '  Parsed request is a module\n' +
      '  using description file: /home/mahler7/green-ext/greencode-ext3/node_modules/python-shell/package.json (relative path: .)\n' +
      "    Field 'browser' doesn't contain a valid alias configuration\n" +
      '    resolve as module\n' +
      "      /home/mahler7/green-ext/greencode-ext3/node_modules/python-shell/node_modules doesn't exist or is not a directory\n" +
      "      /home/mahler7/green-ext/greencode-ext3/node_modules/node_modules doesn't exist or is not a directory\n" +
      '      looking for modules in /home/mahler7/green-ext/greencode-ext3/node_modules\n' +
      '        single file module\n' +
      '          using description file: /home/mahler7/green-ext/greencode-ext3/package.json (relative path: ./node_modules/child_process)\n' +
      '            no extension\n' +
      "              Field 'browser' doesn't contain a valid alias configuration\n" +
      '              /home/mahler7/green-ext/greencode-ext3/node_modules/child_process is not a file\n' +
      '            .js\n' +
      "              Field 'browser' doesn't contain a valid alias configuration\n" +
      "              /home/mahler7/green-ext/greencode-ext3/node_modules/child_process.js doesn't exist\n" +
      '            .json\n' +
      "              Field 'browser' doesn't contain a valid alias configuration\n" +
      "              /home/mahler7/green-ext/greencode-ext3/node_modules/child_process.json doesn't exist\n" +
      '            .wasm\n' +
      "              Field 'browser' doesn't contain a valid alias configuration\n" +
      "              /home/mahler7/green-ext/greencode-ext3/node_modules/child_process.wasm doesn't exist\n" +
      '        existing directory /home/mahler7/green-ext/greencode-ext3/node_modules/child_process\n' +
      '          using description file: /home/mahler7/green-ext/greencode-ext3/node_modules/child_process/package.json (relative path: .)\n' +
      '            using description file: /home/mahler7/green-ext/greencode-ext3/package.json (relative path: ./node_modules/child_process)\n' +
      '              no extension\n' +
      "                Field 'browser' doesn't contain a valid alias configuration\n" +
      '                /home/mahler7/green-ext/greencode-ext3/node_modules/child_process is not a file\n' +
      '              .js\n' +
      "                Field 'browser' doesn't contain a valid alias configuration\n" +
      "                /home/mahler7/green-ext/greencode-ext3/node_modules/child_process.js doesn't exist\n" +
      '              .json\n' +
      "                Field 'browser' doesn't contain a valid alias configuration\n" +
      "                /home/mahler7/green-ext/greencode-ext3/node_modules/child_process.json doesn't exist\n" +
      '              .wasm\n' +
      "                Field 'browser' doesn't contain a valid alias configuration\n" +
      "                /home/mahler7/green-ext/greencode-ext3/node_modules/child_process.wasm doesn't exist\n" +
      '              as directory\n' +
      '                existing directory /home/mahler7/green-ext/greencode-ext3/node_modules/child_process\n' +
      '                  using description file: /home/mahler7/green-ext/greencode-ext3/node_modules/child_process/package.json (relative path: .)\n' +
      '                    use ./index.js from main in package.json\n' +
      '                      using description file: /home/mahler7/green-ext/greencode-ext3/node_modules/child_process/package.json (relative path: ./index.js)\n' +
      '                        no extension\n' +
      "                          Field 'browser' doesn't contain a valid alias configuration\n" +
      "                          /home/mahler7/green-ext/greencode-ext3/node_modules/child_process/index.js doesn't exist\n" +
      '                        .js\n' +
      "                          Field 'browser' doesn't contain a valid alias configuration\n" +
      "                          /home/mahler7/green-ext/greencode-ext3/node_modules/child_process/index.js.js doesn't exist\n" +
      '                        .json\n' +
      "                          Field 'browser' doesn't contain a valid alias configuration\n" +
      "                          /home/mahler7/green-ext/greencode-ext3/node_modules/child_process/index.js.json doesn't exist\n" +
      '                        .wasm\n' +
      "                          Field 'browser' doesn't contain a valid alias configuration\n" +
      "                          /home/mahler7/green-ext/greencode-ext3/node_modules/child_process/index.js.wasm doesn't exist\n" +
      '                        as directory\n' +
      "                          /home/mahler7/green-ext/greencode-ext3/node_modules/child_process/index.js doesn't exist\n" +
      '                    using path: /home/mahler7/green-ext/greencode-ext3/node_modules/child_process/index\n' +
      '                      using description file: /home/mahler7/green-ext/greencode-ext3/node_modules/child_process/package.json (relative path: ./index)\n' +
      '                        no extension\n' +
      "                          Field 'browser' doesn't contain a valid alias configuration\n" +
      "                          /home/mahler7/green-ext/greencode-ext3/node_modules/child_process/index doesn't exist\n" +
      '                        .js\n' +
      "                          Field 'browser' doesn't contain a valid alias configuration\n" +
      "                          /home/mahler7/green-ext/greencode-ext3/node_modules/child_process/index.js doesn't exist\n" +
      '                        .json\n' +
      "                          Field 'browser' doesn't contain a valid alias configuration\n" +
      "                          /home/mahler7/green-ext/greencode-ext3/node_modules/child_process/index.json doesn't exist\n" +
      '                        .wasm\n' +
      "                          Field 'browser' doesn't contain a valid alias configuration\n" +
      "                          /home/mahler7/green-ext/greencode-ext3/node_modules/child_process/index.wasm doesn't exist\n" +
      "      /home/mahler7/green-ext/node_modules doesn't exist or is not a directory\n" +
      "      /home/mahler7/node_modules doesn't exist or is not a directory\n" +
      "      /home/node_modules doesn't exist or is not a directory\n" +
      "      /node_modules doesn't exist or is not a directory",
    stack: "ModuleNotFoundError: Module not found: Error: Can't resolve 'child_process' in '/home/mahler7/green-ext/greencode-ext3/node_modules/python-shell'\n" +
      '    at /home/mahler7/green-ext/greencode-ext3/node_modules/webpack/lib/sharing/ConsumeSharedPlugin.js:179:12\n' +
      '    at finishWithoutResolve (/home/mahler7/green-ext/greencode-ext3/node_modules/enhanced-resolve/lib/Resolver.js:296:11)\n' +
      '    at /home/mahler7/green-ext/greencode-ext3/node_modules/enhanced-resolve/lib/Resolver.js:362:15\n' +
      '    at /home/mahler7/green-ext/greencode-ext3/node_modules/enhanced-resolve/lib/Resolver.js:410:5\n' +
      '    at eval (eval at create (/home/mahler7/green-ext/greencode-ext3/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)\n' +
      '    at /home/mahler7/green-ext/greencode-ext3/node_modules/enhanced-resolve/lib/Resolver.js:410:5\n' +
      '    at eval (eval at create (/home/mahler7/green-ext/greencode-ext3/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:27:1)\n' +
      '    at /home/mahler7/green-ext/greencode-ext3/node_modules/enhanced-resolve/lib/DescriptionFilePlugin.js:87:43\n' +
      '    at /home/mahler7/green-ext/greencode-ext3/node_modules/enhanced-resolve/lib/Resolver.js:410:5\n' +
      '    at eval (eval at create (/home/mahler7/green-ext/greencode-ext3/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)',
    compilerPath: undefined
  }
]

Previously I had been getting error messages like this one below referencing packages that were not in node_modules, but upon installing them it kept mentioning other packages that were missing. I tried creating a polyfill, but wasn’t sure where to add the fallback or how to use the non-polyfill method. Inserting resolve.fallback: { "os": false } at the places mentioned produced syntax errors. Finally after installing all the packages it couldn’t find, I got the error message above mentioning child_process which was already installed and in node_modules. (This message below is from a run on my Mac, hence the different file paths. The kind of error was identical to the one I was getting on my PC though)

ModuleNotFoundError: Module not found: Error: Can't resolve 'os' in '/Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/python-shell'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
        - add a fallback 'resolve.fallback: { "os": require.resolve("os-browserify/browser") }'
        - install 'os-browserify'
If you don't want to include a polyfill, you can use an empty module like this:
        resolve.fallback: { "os": false }
    at /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/webpack/lib/Compilation.js:1762:28
    at /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/webpack/lib/NormalModuleFactory.js:730:13
    at eval (eval at create (/Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:10:1)
    at /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/webpack/lib/NormalModuleFactory.js:273:22
    at eval (eval at create (/Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:9:1)
    at /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/webpack/lib/NormalModuleFactory.js:402:22
    at /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/webpack/lib/NormalModuleFactory.js:117:11
    at /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/webpack/lib/NormalModuleFactory.js:646:24
    at /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/webpack/lib/NormalModuleFactory.js:800:8
    at /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/webpack/lib/NormalModuleFactory.js:920:5
resolve 'os' in '/Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/python-shell'
  
  
  Parsed request is a module
  using description file: /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/python-shell/package.json (relative path: .)
    Field 'browser' doesn't contain a valid alias configuration
    resolve as module
    
    
      /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/python-shell/node_modules doesn't exist or is not a directory
      
      
      /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/node_modules doesn't exist or is not a directory
      
      looking for modules in /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules
        single file module
          using description file: /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/package.json (relative path: ./node_modules/os)
            no extension
              Field 'browser' doesn't contain a valid alias configuration
              /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/os doesn't exist
            .js
              Field 'browser' doesn't contain a valid alias configuration
              /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/os.js doesn't exist
            .json
              Field 'browser' doesn't contain a valid alias configuration
              /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/os.json doesn't exist
            .wasm
              Field 'browser' doesn't contain a valid alias configuration
              /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/os.wasm doesn't exist
        /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/os doesn't exist
      /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/node_modules doesn't exist or is not a directory
      /Users/trevorvillwock1/Desktop/Texas_State/node_modules doesn't exist or is not a directory
      /Users/trevorvillwock1/Desktop/node_modules doesn't exist or is not a directory
      /Users/trevorvillwock1/node_modules doesn't exist or is not a directory
      /Users/node_modules doesn't exist or is not a directory
      /node_modules doesn't exist or is not a directory
/Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/@jupyterlab/builder/lib/build-labextension.js:100
            throw new Error(err.details);
            ^

Error: resolve 'os' in '/Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/python-shell'
  Parsed request is a module
  using description file: /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/python-shell/package.json (relative path: .)
    Field 'browser' doesn't contain a valid alias configuration
    resolve as module
      /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/python-shell/node_modules doesn't exist or is not a directory
      /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/node_modules doesn't exist or is not a directory
      looking for modules in /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules
        single file module
          using description file: /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/package.json (relative path: ./node_modules/os)
            no extension
              Field 'browser' doesn't contain a valid alias configuration
              /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/os doesn't exist
            .js
              Field 'browser' doesn't contain a valid alias configuration
              /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/os.js doesn't exist
            .json
              Field 'browser' doesn't contain a valid alias configuration
              /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/os.json doesn't exist
            .wasm
              Field 'browser' doesn't contain a valid alias configuration
              /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/os.wasm doesn't exist
        /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/os doesn't exist
      /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/node_modules doesn't exist or is not a directory
      /Users/trevorvillwock1/Desktop/Texas_State/node_modules doesn't exist or is not a directory
      /Users/trevorvillwock1/Desktop/node_modules doesn't exist or is not a directory
      /Users/trevorvillwock1/node_modules doesn't exist or is not a directory
      /Users/node_modules doesn't exist or is not a directory
      /node_modules doesn't exist or is not a directory
    at compilerCallback (/Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/@jupyterlab/builder/lib/build-labextension.js:100:19)
    at /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/@jupyterlab/builder/lib/build-labextension.js:142:21
    at /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/neo-async/async.js:2830:7
    at done (/Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/neo-async/async.js:2865:11)
    at /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/neo-async/async.js:2818:7
    at /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/webpack/lib/HookWebpackError.js:69:3
    at Hook.eval [as callAsync] (eval at create (/Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:15:1)
    at Hook.CALL_ASYNC_DELEGATE [as _callAsync] (/Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/tapable/lib/Hook.js:18:14)
    at Cache.shutdown (/Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/webpack/lib/Cache.js:150:23)
    at /Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/webpack/lib/Compiler.js:1124:15
An error occured.
subprocess.CalledProcessError: Command '['node', '/Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master/node_modules/@jupyterlab/builder/lib/build-labextension.js', '--core-path', '//anaconda3/envs/greencode-ext3/lib/python3.9/site-packages/jupyterlab/staging', '/Users/trevorvillwock1/Desktop/Texas_State/GreenCode/greencode_ext3-master', '--development']' returned non-zero exit status 1.
See the log file for details:  /var/folders/2s/1q1ft5c53417qr07dbj0f_mm0000gp/T/jupyterlab-debug-flial7ou.log
error Command failed with exit code 1.
2

Apologies if I am misunderstanding, but it’s not clear to me how you would use the child_process module in this context. The index.ts runs in the browser (after transpilation). child_process is a node module - at least that is my understanding.

1 Like
3

Generally when I want to run something that is Python, I create an extension that sends a HTTP request to the backend (Python) part of the extension and I run the Python there.

4

Are you running a node application from the backend? I wasn’t able to access the repo - it indicated that it required LDAP authentication.

5

Thanks for the reply! Yes, I believe trying to run this in the browser may be my point of confusion here–I read somewhere that child_process can’t be used from the browser but I thought it might be different when jupyterlab is involved since nodejs was included in the conda environment for the extension tutorial I’m basing this on (Extension Tutorial — JupyterLab 3.0.16 documentation). Sounds like that’s for the server-side code though. For this project I’m not using child_process directly, only python-shell, but the compiler still complains about an issue with child_process for some reason, I think since it’s a dependency of python-shell.

There will be a node server running in the backend for this project eventually to facilitate measuring the power usage of code (the eventual goal of the extension), but it’s not integrated with this index.ts file yet.

Sorry about the github issue–it’s an enterprise account through my university so that’s probably the issue. Here’s the index.ts file:

import {
  JupyterFrontEnd,
  JupyterFrontEndPlugin,
} from '@jupyterlab/application';

import { ITranslator } from '@jupyterlab/translation';

import { ToolbarButton } from "@jupyterlab/apputils";
import { DocumentRegistry } from "@jupyterlab/docregistry";
import { INotebookModel, NotebookPanel, INotebookTracker } from "@jupyterlab/notebook";

import { IDisposable } from "@lumino/disposable";

const PythonShell = require('python-shell');

export class ButtonExtension implements DocumentRegistry.IWidgetExtension<NotebookPanel, INotebookModel> {
    
  constructor(app: JupyterFrontEnd) {
      this.app = app;
  }
    
  readonly app: JupyterFrontEnd;

  createNew(panel: NotebookPanel, context: DocumentRegistry.IContext<INotebookModel>): IDisposable {
 
     // Create the toolbar button
     let mybutton = new ToolbarButton({
         label: 'Measure Power Usage',
         onClick: () => {
             PythonShell.run('ipynb_conv2.py', undefined, function (err: any)              {
                 if (err) throw err;
                 console.log('finished');
             });
         }
    });
           
    // Add the toolbar button to the notebook toolbar
    panel.toolbar.insertItem(10, 'MeasureEnergyUsage', mybutton);
    console.log("MeasEnerUsage activated");

    // The ToolbarButton class implements `IDisposable`, so the
    // button *is* the extension for the purposes of this method.
    return mybutton;
  }
}
/**
 * Initialization data for the greencode-ext3 extension.
 */
const yourPlugin: JupyterFrontEndPlugin<void> = {
  id: '@greencode/measureenergy',
  autoStart: true,
  requires: [ITranslator, INotebookTracker],
  activate: (app: JupyterFrontEnd) => {
    const your_button = new ButtonExtension(app);
    app.docRegistry.addWidgetExtension('Notebook', your_button);
  }
}

export default yourPlugin;
6

It is my understanding that you may be able to communicate with the node server either directly from the front end extension that is running in the browser. Or, you may be able to communicate with it from the backend extension that is ran by the Jupyter Server. When you run the cookiecutter, it will set up the infrastructure for building the backend (Python) extension if you indicate that your extension has a server part. The instructions for programming the backend part of the extension are here: Welcome! — Jupyter Server 1.9.0.dev0 documentation

1 Like
7

Here is an example: extension-examples/server-extension at master · jupyterlab/extension-examples · GitHub

8

I think a rationale for including node is that we use it to run npm, which we use for package management.

9

This sounds like the right path to go down, thank you!

10

If you communicate with the node server directly from the browser, then you will need to ensure all the CORS stuff is set up properly. However, if you do it from the backend, then CORS probably isn’t relevant.