Im trying to implement a jupyter notebook as an excercises book.
I want to upload some python files, without certain classes, the scope of this notebook is to write some classes. I do not know how to overwrite the code that I write in notebook and use it in the script.
EXAMPLE:
Python Script:
Class Network:
# Class to implement
class Main:
def init(self):
#code
Jupyter Cell:
Class Network:
# code writing from students
thx for the help…
If I’m understanding correctly, one option is that you’d run the script in the same kernel namespace as the active notebook. For example, imagine you are working in the Jupyter Notebook and you have defined some variables. You also have a Python script called my_script.py.
After running the first few cells to define some variables, you can enter in the Jupyter Notebook cell the following:
%run -i my_script.py
Read about the interactive flag for %run here.
What you’ll have is the classes in your script now active in the same namespace as where you started the notebook already. So now the namespace has what you executed first in the notebook and then along with whatever the script defined and assigned.
Then you can run the next cell in the notebook to replace (‘clobber’) your class from the script and experiment with versions of it.
There’s also the %load magic that would allow you to load into the notebook, specific lines form your source script, see about the -r flag here.
You may need to involve use reload if there’s imports, see here and here. What you describe wouldn’t necessarily but you could actually approach all this with using imports if you prefer and then you’d definitely need it. Not realizing that import only happens once without forcing a reload or restarting the kernel sometimes trips new Jupyter users up.
There’s all sorts of alternatives to do it fancier, for example, you could image actual using nbformat to grab the code from a cell upon saving the notebook and replace the corresponding block in the script with related Python magic, but what I mentioned earlier are the usual types of approaches, I think you may be looking for.
Plus since you are working with classes, there’s decorators but that is probably down the road.
I tried with the interactive flag, this method appers good but doesnt work for me.
I have a big project, with a python script that call all other python files. I want to remove some codes in the lowest python files for training my students.
The problem is that the -i flag doesnt add these portion of code or I did wrong idk.
To explain better:
Maybe i’m wrong the approch. Thanks for help
I think you need to make a toy example. It can be very simple yet have the structure you reference. But it should do something and actually all run. Put it in a gist at Github or a GitHub repo and describe how you expect to run it all. Your first post had pseudocode and I’m not understanding how what I provided doesn’t do what you ask? I cannot tell from your reply. It ‘doesn’t work for me’ isn’t describing the deficiency clearly.
I created a repo on githhub to explain better to you
So, if u go in the notebook u see a class that the student have to write.
This class is called from the script “DQN_PT.py” in BasicRL/algos/ here
So in that notebook, after writing the class i want to test my result, running %run -i example but I have the error NO CLASS “NETWORK” exist
This is only one part of my project. Thanks
Thanks for sharing that. I think I get it now. You were missing incorporating one other convenience of Jupyter/IPython, I think.
But first there were some hiccups deciphering what you shared.
A) I’m not seeing what you say there? Where is that error? It seems the notebook was running and you interrupted it with your Keyboard.
B) This isn’t ‘toy’ example because it seems to rely on a lot of complex packages. Overlooking that to try it anyway since I think I got most of them (thanks to here doing the heavy lifting with all but gym), I still hit a different error than you report. I see ModuleNotFoundError: No module named 'NetworkBuilt':
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
~/Explanation/example.py in <module>
14 # # Run PPO Algorithm
15 learner = BasicRL("DQN_PT", gym_env=env, verbose=2, gamma=0.99, exploration_decay=0.99)
---> 16 learner.learn(1000)
17 #
18 # # # Run PPO Algorithm
~/Explanation/BasicRL/BasicRL.py in learn(self, ep_step)
50 if self.algorithm == "mcPPO": self._run_mcPPO(ep_step)
51 if self.algorithm == "DDPG": self._run_DDPG(ep_step)
---> 52 if self.algorithm == "DQN_PT": self._run_DQN_PT(ep_step)
53 if self.algorithm == "DQN": self._run_DQN(ep_step)
54 if self.algorithm == "TD3": self._run_TD3(ep_step)
~/Explanation/BasicRL/BasicRL.py in _run_DQN_PT(self, ep_step)
140
141 def _run_DQN_PT(self, ep_step):
--> 142 from BasicRL.algos.DQN_PT import DQN
143 assert (self.discrete_env), "DQN requires discrete environments!"
144 # Iniziallizza Valori
~/Explanation/BasicRL/algos/DQN_PT.py in <module>
6 import gym
7 import random
----> 8 from NetworkBuilt import Network
9
10
ModuleNotFoundError: No module named 'NetworkBuilt'
So given that error, in particular coming from the line from NetworkBuilt import Network in the script DQN_PT.py, I think you intend to make a file NetworkBuilt.py and insert your Network code there, right?
So change your first code cell in your DQN.ipynb file to add one line to the top, like the following:
%%writefile "NetworkBuilt.py"
import torch.nn as nn
# Create the Mode
class Network(nn.Module):
def __init__(self, input_shape, output_size, hiddenNodes=32):
# Input -> 64 -> 64 -> output
super(Network, self).__init__()
self.input_layer = nn.Linear(in_features=input_shape, out_features=hiddenNodes)
self.hidden = nn.Linear(in_features=hiddenNodes, out_features=hiddenNodes)
self.hidden2 = nn.Linear(in_features=hiddenNodes, out_features=hiddenNodes)
self.output_layer = nn.Linear(in_features=hiddenNodes,
out_features=output_size) # np.array(output_size).prod())
def forward(self, x):
# x = nn.functional.relu(self.input_layer(x))
x = self.input_layer(x)
x = nn.functional.relu(self.hidden(x))
x = nn.functional.relu(self.hidden2(x))
return self.output_layer(x)
It is just adding that first line to write out the code your students will draft in that cell. %%writefile is a convenient IPython magic that Jupyter inherits, see here.
You’ll even see it makes the output Overwriting NetworkBuilt.py when you run that cell, which fits nicely with the title of your post.
Importantly, I tested with a fresh clone of your repo and just adding that one line to the top of the first code cell makes your current notebook work.
Importantly, when your students iterate on adjusting the code for the class Network(), you should emphasize to run Kernel > Restart and Run All Cells (or the equivalent in your version of Jupyter). Otherwise, the import won’t get renewed and so just exectuing %run -i example.py will result in using the already imported version of NetworkBuilt.py being used. Python doesn’t like to import over and over as a time saving step since imports may be encountered multiple times. However, while that is convenient for normal users of a package, this can be a cause of dismay when coding/developing unless you know how to force it to use the code that has been updated.
Does that work as an acceptable approach for teaching?
Addendum about scope and namespace
What you originally had without the %%writefile() step does work somewhat as I said. (Assume for this section you comment out the line from NetworkBuilt import Network in the script DQN.py.) For example, if you add print(dir()) to the top of example.py and in the main indent there, you’ll see Network is listed as a defined object. That is because of the -i flag use in the %run command that I described. (Note using the -i flag still turned out to be important to get the run information to show up in the notebook as it runs. Without it, things don’t display until the script completes or is interrupted it seems.) I am not seeing why the scope in Basic.RL.py and DQN.py is very different and does not include that. When I put print(dir()) in DQN.py script or put a variable I defined in the notebook in Basic.RL.py, I see something very different for the objects available when I run DQN.ipynb. I’m not understanding where this complexity is coming from. I’m guessing it may be coming from gym, specifcally gym_env=env, but who knows at this point and I don’t have time to investigate further right now. So I guess it is good you didn’t stick a toy example. It is a long way of saying that I think normally what you had showed would have worked if the line from NetworkBuilt import Network was removed from the script DQN.py; however, something isn’t exactly normal in this complex example.
You perfectly found what I need. Thanks for your help.
Answering on the last question, I’m not a Teacher, I have to do this to finish my internship (in the university). It could work.