Running Jupyter Notebooks Remotely on Poseidon#
Colette Kelly · October 2, 2023 · how-to, computing
Based on this blog post from the Alexander Lab.
1: Connect to Poseidon#
Open a Terminal window on your local machine and type:
colette$ ssh <whoi.ID>@poseidon.whoi.edu
2. Start interactive session on a compute node#
Replace -N 1 with how many nodes you’re requesting, -n 16 with how many cores per node (up to 36), and --mem=16GB with how much memory per core (up to 192GB). The session will time out if you go over the requested memory. Replace --time=00:30:00 with how much time you need.
[colette.kelly@poseidon-l1 ~]$ srun -N 1 -n 16 --mem=1000 --time=00:30:00 --pty bash
To list how much memory is available on each node, run:
sinfo -N -l
3. Module load miniconda#
To use conda or any other software on Poseidon, you need to load the associated module first.
[colette.kelly@pn023 ~]$ module load miniconda/23.11
4. Initialize conda environment#
[colette.kelly@pn023 ~]$ . $CONDA_PREFIX/etc/profile.d/conda.sh
5. Activate conda environment#
Replace envname with the name of your environment:
[colette.kelly@pn023 ~]$ conda activate envname
6. Launch Jupyter Notebook without a browser window#
(envname) [colette.kelly@pn023 ~]$ jupyter notebook --no-browser --port=8888
You’ll need to use a port number that isn’t already in use. To check which ports are already in use, run:
ss -tuln
To launch Jupyter Lab instead, the prompt is almost exactly the same:
(envname) [colette.kelly@pn023 ~]$ jupyter lab --no-browser --port=8888
If your Jupyter Notebook isn’t loading in your browser window, try:
(envname) [colette.kelly@pn023 ~]$ jupyter server --no-browser --port=8890 --ServerApp.default_url=/tree
7. Create an SSH tunnel#
Open a new Terminal window on your local machine and type the following. Replace colette.kelly with your username. Replace pn023 with whatever node the interactive session is running on.
(base) CowardlyLion:~ colette$ ssh -t -t colette.kelly@poseidon.whoi.edu -L 8888:localhost:8888 ssh pn023 -L 8888:localhost:8888
8. Open Jupyter Notebook in a browser window#
Copy-past the url that begins with “http://localhost:8888/tree?token=” into a browser window. Or,
if Jupyter passwork is configured, just enter localhost:8888 in a browser window.
You can set configure a Jupyter password by entering the following in a terminal:
jupyter notebook password
You’ll need to restart your Jupyter Notebook session to be able to use the new password.
9. Shut down Jupyter Notebook#
Make sure you shut down the Jupyter Notebook (either control-C in Terminal or with the Jupyter GUI), otherwise you might run into issues re-opening a port.
Troubleshooting#
If, in creating the SSH tunnel, you get the error:
bind: Address already in use
1. Shut down the existing notebook (either ctrl-C. in Terminal or with the Jupyter GUI). Disconnect the ssh tunnel with ‘logout’ and ctrl-C.#
2. In the first terminal window, start a new notebook with a different port#
You can do this by replacing 8888 with a different number. E.g.,
(envname) [colette.kelly@pn023 ~]$ jupyter notebook --no-browser --port=8889
3. In the second terminal window, start a tunnel to the new port#
Replace 8888 with the new number, e.g.:
(base) CowardlyLion:~ colette$ ssh -t -t colette.kelly@poseidon.whoi.edu -L 8889:localhost:8889 ssh pn023 -L 8889:localhost:8889
4. Open Jupyter Notebook in a browser window#
You can either copy the notebook URL or enter localhost:8889 in a browser window, making sure the number matches the port number from above.
You may need to iterate through 1-4 above a few times until you find an open port.