Using Jupyter on your cluster

Using Jupyter on your cluster#

Setup jupyter#

Install jupyterlab and nb_conda_kernels using conda on the cluster.

I recommend installing them into their own environment. Let’s call it jupyter.
```
conda create -y -n jupyter -c conda-forge jupyterlab nb_conda_kernels
```
But if you’re not concerned about a package in your base environment conflicting with jupyter or its dependencies now or in the future, you can just install them in your base environment, instead.
```
conda install -y -n base -c conda-forge jupyterlab nb_conda_kernels
```
If you installed jupyter into its own environment (as recommended), you’ll need to activate that environment every time you want to use it. Do this now.
```
conda activate jupyter
```

jupyter server --generate-config
jupyter server password

Setup a conda environment to use with jupyter#

Scenario: You would like to install R version 4.0 and use it from within Jupyter. In order to access your new R installation from within Jupyter, you will also need to install IRKernel. So, create a new environment called r4 containing both R version 4.0 and IRKernel:

conda create -y -n r4 -c conda-forge 'r-base=4.0' 'r-irkernel'

Scenario: A similar strategy can be employed for Python. Let’s say you’d like to install python 3.5 in a new conda environment called py35, and you want to run your python 3.5 installation from within Jupyter. To do this, just install both python 3.5 and IPython kernel in a new environment:

conda create -y -n py35 -c conda-forge 'python=3.5' ipykernel

Scenario: Instead, you have an existing environment called py35 with python 3.5, and you’d like to run your python 3.5 installation from within Jupyter. Just install IPython kernel in the existing environment:

conda install -y -n py35 -c conda-forge ipykernel

If you want to install a python package to use within your notebooks, you can install it in the py35environment, for example. You should never need to install anything in the jupyter environment!

In general, you will need to install a new kernel in every conda environment that you would like to use with jupyter. But you do not need to install a new copy of jupyter in every environment. The kernels that you install will allow jupyter in your jupyter environment to execute each version of Python and R that you install in other environments. And the nb_conda_kernels package will automatically detect kernels you’ve installed in those environments on startup. Each kernel should appear when you start up jupyter:

Usage#

When ssh-ing into the cluster, make sure to also add a port-forward to your ssh command. This will connect a port on your computer (ex: 8888) to a port on the cluster (ex: 9999):
```
ssh -L 8888:localhost:9999 your_server_username@your_server_address
```
It’s possible for you to choose a port that is being used by somebody else on the cluster. Pick a port 1024 < x < 65535 that nobody else is using. So don’t use 9999.
If you installed jupyter into its own environment, activate that environment on the cluster:
```
conda activate jupyter
```
On the cluster, start jupyter with the cluster port you used in step #1. And don’t let jupyter try to open a browser:
```
jupyter lab --no-browser --port 9999
```
In a browser running on your own computer, go to http://localhost:8888/lab
Log in using the password you created in step #3 of the setup
Your R and python installations/environments should appear in the Notebook Launcher menu. Click on one to create a notebook. Watch this video to learn how to write one.

When you’re done, make sure to shut down both your notebook kernel(s) (“Kernel” > “Shut Down All Kernels”) and Jupyter itself (“File” > “Shutdown”) from the menu bar! Otherwise, they might continue running on the cluster forever, especially if you use screen sessions.

Usage on an HPC#

If you are working on a high performance computing cluster like TSCC or Expanse, you should perform the following steps between steps 1 and 2 from above.

Start an interactive node on your computing cluster
Reverse-forward the port you chose (from step #1 above) from your interactive node back to the login node. Run the following on the interactive node:
```
ssh -NfR 9999:localhost:9999 $SLURM_SUBMIT_HOST
```
$SLURM_SUBMIT_HOST will automatically determine your login node (ex: login1, login2, etc).

You’re done! The rest of this document is optional.

Simplified Usage Setup (optional)#

It’s possible to reduce the usage instructions above to just three memorable steps (see the next section below). Just follow these recommendations for a bit of extra setup.

Remove the port-forwarding step#

You can set up ssh so that it creates a port forward by default, whenever you connect to the cluster. Just add the LocalForward directive to the ~/.ssh/config on your computer like this. Here’s an example of what you can put in your ~/.ssh/config:

Host a_short_server_nickname
    HostName your_server_address
    User your_server_username
    LocalForward 8888 localhost:9999

If you like to ssh into your cluster multiple times from different terminals, you should also add the following to the top of your ~/.ssh/config, so that ssh will reuse the same connection.

Host *
    ControlMaster auto
    ControlPath ~/.ssh/ssh_mux_%h_%p_%r

Remove the arguments that you pass to Jupyter#

You can have jupyter use the arguments above by default, whenever it is executed. Just set them in the ~/.jupyter/jupyter_server_config.py config file, similar to what they do here:

Change the default port from this

## The port the server will listen on (env: JUPYTER_PORT).
#  Default: 0
# c.ServerApp.port = 0

to this

## The port the server will listen on (env: JUPYTER_PORT).
#  Default: 0
c.ServerApp.port = 9999

Don’t open a browser. This should be False by default:

## Whether to open in a browser after starting.
#                          The specific browser used is platform dependent and
#                          determined by the python standard library `webbrowser`
#                          module, unless it is overridden using the --browser
#                          (ServerApp.browser) configuration option.
#  Default: False
# c.ServerApp.open_browser = False

Do other things in your terminal while Jupyter is running#

To prevent jupyter from barfing into your terminal when it starts running, you can send the start-up command to the background and silence it in step #3 of the usage section:

jupyter lab &>/dev/null &

Activate the jupyter environment and execute jupyter simultaneously#

For those who installed jupyter into a separate jupyter conda environment, you can actually activate the environment and execute the command simultaneously, in a single command:

conda run -n jupyter jupyter lab

Create an alias#

You can also create an alias called jup so that you don’t have to type out the full command to start jupyter every time. Just run this command to add the alias to your .bashrc file on the cluster:

echo 'alias jup="conda run -n jupyter jupyter lab &>/dev/null"' >> ~/.bashrc

Then, close and reopen your terminal or ssh connection.

Simplified Usage (optional)#

Assuming you followed the steps to setup the simplified usage, the usage instructions will reduce to just three (or four) memorable steps:

ssh into the cluster
```
ssh your_short_server_nickname
```
If you are working on an HPC, you can login to an interactive node and reverse-forward your port at this stage.
Start jupyter on the cluster
```
jup &
```
In a browser running on your own computer, go to http://localhost:8888/lab

Use screen to keep Jupyter running even with bad internet!#

If your ssh connection dies because of a choppy internet connection, your Jupyter instance will die, too. You can run jupyter from within a screen session to make it persist. Let’s adapt the Simplified Usage instructions.

ssh into the cluster
```
ssh your_short_server_nickname
```
Create and enter a screen session named jupyter
```
screen -S jupyter
```
View a list of running screen sessions
```
screen -ls
```
If you are working on an HPC, you can login to an interactive node and reverse-forward your port at this stage.
Start jupyter on the cluster
```
jup &
```
In a browser running on your own computer, go to http://localhost:8888/lab
If your ssh connection dies, everything in your screen session will continue running. Just log back in and reattach the session to continue where you left off. (Note: If you are working on an HPC, you will need to use the same login node as before!)
```
screen -r jupyter
```
You can also terminate a session while inside of it
```
exit
```
Or, if you’d like to detach from a session (instead of terminating it) while you’re inside of it, press ctrl+a and then the letter d