How are you monitoring your memory usage?
If you are using JupyterLab, there is a nice monitor that is part of the status bar along the bottom. See Mem: along the bottom of this gif to get an idea of where to look.
If you are using the classic notebook interface maybe try %memit magic, see here? Or nbresuse? Or try some profiling?