OK. . . now what? (AKA, What do I do with this "Jupyter" thing anyway?)

General
1

Greetings!

This has to be the ULTIMATE newbie/neo/nugget question around here. . .

A note about myself:
I’ve been around the block at least once. Maybe even twice! I know which end of the keyboard faces upwards, and I know where the “any” key is. Though I may not be able to code with both feet while sleeping, I generally can tell the difference between a Python file and something written in C by visual inspection.

Visual Studio/Visual Studio Code, Thonny, UltraEdit, and other editors/IDE’s I am familiar with, and can generally figure my way around them.

The first time I ever knew that Jupyter existed was after I bought a GoPiGo robot

The 'bot came with a simple interface that could use a Blockly-type coding interface, or Python via Jupyter.

Jupyter?  What the. . .??!!

Given the following illustration:

Jupytr wassit
Jupytr wassit1897×847 180 KB

Like I said, I’ve used traditional IDE’s before. Here? I have no clue.

I’ve gone to the Jupyter web-site. (Obviously. . . . Otherwise I wouldn’t be HERE.)

There’s lots of stuff there all about how Jupyter is the Next Best Thing since Peanut Butter and Jelly, how the scientific community loves it, and some interesting illustrations of saddle-curves and spinning nebulae. . . .

Blast it all, I just want to write some stinkin’ code!  Fuzz THIS!!  I got better things to do with my life than. . .

So. . .

Where do I start?

What I haven’t seen is something that says "OK, here’s how to work the beastie. Here’s a nice step-by-step tutorial on how to code in “Jupyter” per. se.

If I were a teacher with an elementary or secondary school class that I had to teach this, and assuming I was expected to actually use this Jupyter stuff as part of the curriculum, where would I even begin?

Not only can I code in this, (supposedly), but I’ve also heard that there is something called a “Jupyter console” that works like a terminal session that I can type commands into, like “sudo such-and-so”, without having to leave the Jupyter session I am in.

Where is it? How do I find it?

I am sure I am making a total jackass out of myself by asking this stuff, but I’ve been looking at this for a while.

  • Programming EEPROMS in 8085 assembly language, by hand, using pencil-and-paper, and typing the resulting hex into a prom-burner one byte at a time?
    Done that.
  • Embedded controller projects for point-of-sale advertising displays?
    Got 'cha covered!
  • Writing programs in Python that will allow my robot to be lead around by a joystick while sending back real-time, first-person video?
    Check!  (Though the JavaScript is a bit nutsy. . .)

Figuring out the Jupyter interface?
Embarrassing. . . . .
:crazy_face: :face_with_symbols_over_mouth:

P.S.
I have seen the Official Documentation, which goes to great lengths to describe the Jupyter environment in excruciating detail.

Trying to use that as a “new user tutorial” is like trying to learn C by reading the POSIX language specification, or learn Python by reading PEP’s.
(insert emoji of smiley banging head on wall)

2

I could equally ask: given an IDE, or IDLE, now what am I supposed to?

Here’s the intro notebook I’ve been drafting as the first notebook a student new to notebooks will see in a course block we’ll be rolling out to students in a month or so.

It does the whole “here’s how the notebook UI” thing, but how you would expect it start?

One of the advantages for education is that the notebook lets you mix instructional guidance with code, a code execution environment and code outputs in a single browser window, rather than presenting the student with a PDF document or VLE webpage that provides you with instructions on how to open an editor / IDE or points you to Youtube videos then tells you to go somewhere else on your computer to open and IDE application, then tells you what buttons to press and what to type where, then refers you to some documentation to read elsewhere, etc.

2 Likes
3

Well…
I remember thinking that the first time I tried to use a “drum control card” on an IBM 029 keypunch too. :wink:

At least for crufty old timer’s like myself, the idea of a “blank slate” - be it a deck of punched cards, a green-screen VT-100 terminal, a display on a PC of days gone by, an open text editor window, or something like Thonny on the Raspberry Pi - makes sense as a programming paradigm.

  • I write code that (hopefully) works.
  • I try executing the code and see what happens.
  • I make any needed corrections.
  • And I keep doing this until I decide I need a break, or the code works.

The Jupyter paradigm seems more like:

  • Here’s a little blip of code that does something simple.
  • Here’s a text blurb that talks about something relating to the code. (hopefully)
  • Here’s another little code blip.
  • Here’s more text, maybe an interactive illustration or two. . . .
  • Followed by (maybe) another little blip of code.
  • And so on. . .

If I wanted to teach something, (like Python, mathematics, science, or something else requiring a bit of self-paced interaction with the student), and do it remotely, (which is what your documents seem to want to do), Jupyter notebooks might be a great interactive teaching tool. (BTW, I like your documents. I plan on re-reading them at greater length later on if you don’t mind.)

However, (for example), if I wanted a classroom of students to develop and program a specific and fairly complicated task using Python, C, Tensor-Flow, (or something else), that would be a programming project involving larger amounts of interrelated code, possibly even code being developed by different teams, I’m not sure how Jupyter would help.

Git? Absolutely!
Jupyter? I’m not so sure.

Let’s say I’m doing a project like the “remote control robot” project I’m working on now. It has two major parts:

  • “Server-side” code that exists, (and runs), on the robot itself and communicates to a remotely connected web browser, accepts responses, and controls the robot accordingly. In this case, Python and some off-the-shelf web-server stuff.

  • “Client-side” code, primarily in CSS, HTML, and JavaScript that will (ultimately) run on the client browser, is used to receive video and status messages from the robot and, (ultimately), is used to send control and command messages back to it.
     
    The whole effect being that of a remotely controlled robot, (like a “bomb-sniffing” robot), that transmits video back to the handler with status information superimposed on it as a “Heads Up Display” type overlay.

How would a project of this size, using a whole host of external libraries and packages brought in from a number of external sources, (look up “nipple.js” if you’re interested), be managed within the context of a Jupyter-based project?

Thanks!

I really appreciate the feedback.

P. S.
Why I ask:

I have been tasked with beta-testing a pre-release version of the latest “classroom” operating system for the GoPiGo robots, and Jupyter is a part of it. I would really like to be able to get my arms around it well enough to provide at least some degree of test coverage.

4

The “are Jupyter notebooks appropriate for developing multi-file code libraries” is a whole other issue. Things like the JupyterLab UI provides a range of UI features that nudge towards “traditional” development environments, and support from VS Code Python extension lets you develop in that environment against Jupyter notebooks. Other editors are also quite happy to hook into Jupyter kernel servers.

For learners / novices writing short, single file programs, the notebook UI perhaps encourages them to make literate / narrative comments in the code.

Using Jupytext, you can also edit / save Python files, markdown files, etc etc, using the notebook UI if that style of UI works for you.

If the classic simple notebook UI is too simple for you, move to JupyterLab (which is what Jupyter core effort is focussed on in terms of UI development, classic notebook having been moved in maintenance mode, from what I can tell).

Personally, I couldn’t care less if “real developers” don’t like notebooks (and I’m happy to debate that!;-), nor if they think it teaches bad habits (similarly…!). It works fine for editing and running the short scripts I create to scratch my own itches, and seems to work pretty well for us in an edu setting. The context I work in also tends to focus on teaching students ideas about programming, not skills about developing (that is a completely separate debate and one I am happy to enter into in another thread;-). Notebooks work for that in many cases.

Also, many use cases (stats analyses, digital humanities scraping tasks, simple map generation) require little more than a short single page script and powerful off the shelf packages to let users create their own end-user scripts to get stuff done. This doesn’t require an IDE, it requires an environment that makes computing environments available to folk who would never consider themselves to be developers, who don’t have the sys admin skills to set up environments, and who would find a traditional IDE over-complicated, overbearing, not helpful and just way too confusing for the simple tasks they want to achieve, like running a single line of code.

–tony

PS there are various tools exploring testing strategies around code in notebooks, eg https://test-book.readthedocs.io/en/latest/ or https://github.com/jpmorganchase/nbcelltests

3 Likes
5

Very well said.

This discussion has really helped me to get a better understanding of what the whole “Jupyter” (at least “notebook”), paradigm is about.

I totally agree.
“Real” developers do whatever is needed with the best tool for the job. “Attitude” > /dev/null. :wink:

On the GoPiGo, I like the “Bloxter” editor, as it lets me test simple concepts in a way that avoids all the pain of building a “real” app just for some “throw-away” code. Especially if the GoPiGo’s sensors or actuators are involved.

So, does that make me a dweeb punter? Like I said, all attitude goes to /dev/null!

The few times I’ve had the opportunity to teach, I always teach that “the best tool for the job” is what works for you.

6

I usually recommend the following video and Github repo from SciPy 2019 (Scientific Computing Conference for Python) where the Jupyter core devs conducted a step by step walkthrough of Jupyter and the features available. I have found it is effective with folks that are completely new to writing code so hopefully it is a help to you as well.

Video: https://www.youtube.com/watch?v=RFabWieskak
Github Repo: https://github.com/jupyterlab/scipy2019-jupyterlab-tutorial#installation

In general, I would say that Jupyter is used more for scientific/data science/experimentation tasks and less often for traditional software development. If you are interested more broadly how folks use Jupyter then I would also suggest the following blog from Netflix on some of the cool stuff they are using notebooks for (and I think does a good job of summarizing the different personas/tasks)
Link: https://netflixtechblog.com/notebook-innovation-591ee3221233

There is some work (nbdev project) on adapting Jupyter to tackling more traditional software development workflows but it is still pretty early from an adoption standpoint.
Link: https://nbdev.fast.ai
Video: https://www.youtube.com/watch?v=Hrs7iEYmRmg

2 Likes
7

Hi,

Yes - I personally use notebooks often for exploring a data file or an idea, and for introductory teaching, but I always use a text editor and the terminal or an IPython session, for software development. I think you’ll find that’s pretty typical.

In fact, many of us who teach with notebooks fret about how to help our students to switch out of notebook mode when their task gets closer to software development - such as refactoring code into libraries and so on.

Cheers,

Matthew

1 Like
8

Excellent!

Thanks for the links, though my wife will probably want to have a few words with you about me staying up all night reading this stuff! (:laughing:)

Maybe this isn’t as bad as it seems?

<possibly unrelated idle ramblings>

Way back in the Dark Ages, back when we had keypunch machines and computers ran on coal, (note that some earlier computers still burned wood, but I digress), I remember the idea of multiple file stream forks fascinated me and I thought it was a great idea.

I imagined a multiply-forked data-stream being especially useful for technical, scientific, or programming tasks since my vision for the multiple forks had one being a “working” fork, (i.e. the actual program code and/or comments), and another being a “resource” fork containing documentation, working thoughts, idle ramblings, references to other docs; things that are important to the code or the project, but that didn’t belong in the specific part of the file that contained the source code or executable binary.

One of the things I’ve wished for when programming is a way to create a source code file that would appear to be a source code file with code, comments, etc. etc. etc. - but have the ability to “unfold” into other documents that relate to a specific part of the code.

I realize that this is a bad description, let me explain:

While coding, I often talk to myself, (It drives my wife NUTS!), because if I can’t describe what I want in words, I don’t yet know enough about it to code it.

Using the “remote control robot” project I mentioned before as an example, I am refactoring the code to use a standard browser joystick API interface to control the robot.

To describe how the position of the joystick controls how the robot moves, I created an (imaginary) grid divided into four quarters, with the imaginary joystick handle in the center of the two intersecting (horizontal and vertical), lines.

To calculate the required wheel-speed ratio for the two wheels, based on the position of the joystick, I had to (mentally) analyze the meaning of particular joystick readings, along with their sign - value is magnitude and sign is direction.

  • Am I moving forward or backwards?
  • Am I moving to the left or to the right?
  • Am I traveling slowly or quickly?
  • What happens when the sign of a joystick input changes?
    (i.e. I move from “forward” to “backward” or from “left” to “right”)
  • How do I calculate the appropriate vector so that I can get the wheel-speed ratios?
    (Here is where I tip my hat to the hell my Trig teacher put me through, it paid off!)

There’s a lot of thought, potential calculation, ideas, scrapped ideas, things I haven’t tried yet, etc., etc., etc., that goes on behind a particular routine/method/function/whatever in the code, and it’s easy to loose track of it.

As a consequence of these thought processes, I tend to “over-comment” my code - eventually having to cut huge sections of comments out and paste them into their own documents.

What I wish for is an IDE/programming environment that would allow me to code, (along with the usual goodies like parenthesis/bracket matching, code completion, linting, etc.), along with the ability to “unfold” into a “scratchpad” linked to a particular part of the code where I could write things that are important to the understanding of that particular part of the code. With this, I could come back to something I wrote months, maybe even years later and be able to “unfold” my thought processes with respect to a part of what I did.

Instead of embedding blurbs of code into some text, or blurbs of text into some code, it would be nice to have this as a monolithic whole. I could write code, propose tests, examine theories, prepare documentation, and have it all tracked in one place.

After all these years I still dream of being able to go into a piece of code, right-click on something, and have it pop-up the breadcrumbs and facts I saved for myself, instead of having to wail and gnash my teeth wondering what I was smoking when I wrote it.

Maybe this is the kind of IDE that Jupyter could become? The transition from “Notebooks” to a “real IDE” could be simpler if they were one and the same.

Then again, maybe this is all just a nice dream, 'eh?

</possibly unrelated idle ramblings>

9

Using Jupytext, which allows you to author in notebook UI and automatically save it in one or more file formats on the back (ipynb, py of various flavours, markdown in various flavours, Rmd, etc), you can tag cells as “active-ipynb” and these are runnable in the notebook UI but are saved into md/py etc files as commented out cells. This means you can have “for example” code that runs in a notebook if you open a py file in the notebook UI using Jupytext, but is commented out in the py file which you might want to use as a simple package file. Docs: https://jupytext.readthedocs.io/en/latest/formats.html#active-and-inactive-cells

3 Likes