Software
Running JupyterLab at HPC2N - extra Python modules
Running JupyterLab at HPC2N - extra Python packages
For how to use Jupyter notebook in general, see the Using Jupyter at HPC2N page.
Already installed at HPC2N
If you need extra Python modules and these modules are already installed at HPC2N, you can just load them. It is easiest to do so before you start the JupyterLab, inside the submit file.
Example, loading JupyterLab/3.2.8, its prerequisites, and the modules for SciPy-bundle (mpi4py, numpy, pandas, scipy etc.) and matplotlib :
#!/bin/bash # Here you should put your own project id #SBATCH -A hpc2nXXXX-YYY # This example use 1 core #SBATCH -n 1 # Ask for a suitable amount of time. Remember, this is the time the Jupyter notebook will be available! HHH:MM:SS. #SBATCH --time=05:00:00 # Clear the environment from any previously loaded modules module purge > /dev/null 2>&1 # Load the module environment suitable for the job module load GCC/10.3.0 JupyterLab/3.2.8 OpenMPI/4.1.1 SciPy-bundle/2021.05 matplotlib/3.4.2 # Start JupyterLab jupyter lab --no-browser --ip $(hostname)
See here for an explanation of the options to jupyter.
With some own-installed Python packages
Running JupyterLab with some own-installed Python packages requires you to use a virtual environment and your own Jupyter kernel. This is not as difficult as it may sound.
Example Own-installed pyDOE and torch. Using JupyterLab/3.4.2
1) First we need to load the modules that we need for our own-installed packages, and for the JupyterLab.
pyDOE and torch has some prerequisites, some of which are already installed at HPC2N. We will start by loading the available prerequisite modules:
module load GCC/10.3.0 JupyterLab/3.2.8 OpenMPI/4.1.1 SciPy-bundle/2021.05 matplotlib/3.4.2
2) We now need to create a virtual environment (venv) to install our own packages in. I am placing it in the Public directory under my home directory ($HOME), but you could instead place it in your project storage. I am calling the venv "jupvenv", but you can call it what you want:
python -m venv $HOME/Public/jupvenv
3) Activate the venv
source $HOME/Public/jupvenv/bin/activate
4) Install ipykernel in the venv. This is needed to be able to make your own Jupyter kernel which can use the own-installed Python packages
pip install --no-cache-dir --no-build-isolation ipykernel
NOTE! It may complain of missing prerequisites. If so, instead install:
pip install --no-cache-dir --no-build-isolation pyparsing pytz jinja2 packaging webencodings cffi babel jsonschema requests tomlkit wheel ipykernel
5) Install your Python packages in the venv, here pyDOE and torch
pip install --no-cache-dir --no-build-isolation pyDOE torch
6) Install the new kernel in Jupyter (here called jupvenv)
python -m ipykernel install --user --name=jupvenv
7) Check list of kernels to see your new kernel
jupyter kernelspec list
Later you can remove the kernel if you feel like, using this:
jupyter kernelspec uninstall jupvenv
8) Now make a submit file as before. Something like this should work:
#!/bin/bash # Here you should put your own project id #SBATCH -A hpc2nXXXX-YYY # Here allocating 1 core - change as suitable for your case #SBATCH -n 1 # Ask for a suitable amount of time. Remember, this is the time the Jupyter notebook will be available! #SBATCH --time=05:00:00 # Clear the environment from any previously loaded modules module purge > /dev/null 2>&1 # Load the module environment suitable for the job module load GCC/10.3.0 JupyterLab/3.2.8 OpenMPI/4.1.1 SciPy-bundle/2021.05 matplotlib/3.4.2 # Activate the venv you installed your own Python packages to source $HOME/Public/jupvenv/bin/activate # Start JupyterLab jupyter lab --no-browser --ip $(hostname)
See here for an explanation of the options to jupyter.
9) Submit the above submit file (here I named it MyJupvenv.sh).
sbatch MyJupvenv.sh
You get the <job-id> when you do the above command.
Check the SLURM output file (slurm-<job.id>.out); grab the URL with the hostname as described in the first part of this document, since the localhost one requires you to login to the compute node.
10) Start a webbrowser within HPC2N (ThinLinc interface). Put in the URL you grabbed, including the token.
11) Inside JupyterLab, start the new kernel. Just click the launcher for that one if no other kernel is running.
If a kernel is running (shown under kernels), then shut down that kernel and click "Kernel" in the menu, and then "Change kernel". Pick your kernel from the drop-down menu.
12) You can now run your files etc. with the own-installed Python packages available.
NOTE! Sometimes it is still running on the default kernel. If so, Click the 3 little dots in the right side of the editor-window for the program and pick your kernel. Then rerun your files.