Dask.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f4f3c810-5c4a-41cc-9eb6-e6d8e549cc63",
   "metadata": {},
   "source": [
    "# Dask demo on Slurm cluster\n",
    "\n",
    "I have [miniconda](https://docs.conda.io/en/latest/miniconda.html) in my home folder, so I used that to set up a conda env,\n",
    "\n",
    "```\n",
    "conda create -n jlabdask python jupyterlab numpy matplotlib -y\n",
    "conda activate jlabdask\n",
    "conda install dask distributed -c conda-forge\n",
    "pip install -U click==8.0.2\n",
    "```\n",
    "To ensure the jupyter lab continues when I'm not connected, I start it in a tmux pane (you could use screen as well)\n",
    "```\n",
    "tmux\n",
    "```\n",
    "then inside tmux\n",
    "```\n",
    "source activate ~/miniconda3/bin/activate\n",
    "conda activate jlabdask\n",
    "jupyter lab\n",
    "```\n",
    "Look at the port number it chooses, then forward it over your SSH connection (replace 4288 by the number jupyter shows)\n",
    "```\n",
    "ssh -L 4288:localhost:4288 me@door\n",
    "```\n",
    "and open the lab URL in your browser, then we can get started"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a1c15a06-9303-434a-bdbb-3df910049193",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using matplotlib backend: <object object at 0x7fe15460a150>\n",
      "%pylab is deprecated, use %matplotlib inline and import the required libraries.\n",
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab\n",
    "import dask, os, time\n",
    "from dask.distributed import Client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6dad57f-b57f-4d9b-9f0c-eda44c684fd2",
   "metadata": {},
   "source": [
    "# Setting up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d25c30a3-ca2b-4bee-bc78-521fa585d1d8",
   "metadata": {},
   "source": [
    "We first need to start a scheduler on door, we can just check we don't have one running already,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f4686005-6943-40d5-a772-e5d5e2c97895",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "marmadu   398999  0.0  0.0  13056  3012 pts/15   Ss   13:48   0:00 /bin/bash -c ps aux | grep dask-sched\n",
      "marmadu   399001  0.0  0.0  12136  1104 pts/15   R    13:48   0:00 grep dask-sched\n"
     ]
    }
   ],
   "source": [
    "!ps aux | grep dask-sched"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "55c5c4bf-3c6c-437c-9df7-ddd28242e09d",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.system('dask-scheduler --host=10.11.7.58 &> scheduler.log &')\n",
    "time.sleep(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "8a1e61c7-78af-49de-8213-01b1d07d9fe7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "distributed.scheduler - INFO - -----------------------------------------------\n",
      "distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy\n",
      "distributed.scheduler - INFO - -----------------------------------------------\n",
      "distributed.scheduler - INFO - Clear task state\n",
      "distributed.scheduler - INFO -   Scheduler at:     tcp://10.11.7.58:8786\n",
      "distributed.scheduler - INFO -   dashboard at:           10.11.7.58:8787\n"
     ]
    }
   ],
   "source": [
    "!tail scheduler.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "6b9fdec4-22f1-4526-b0d7-9028b9d576c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "marmadu   409653  5.2  0.1 464728 115128 ?       Sl   15:06   0:01 /home/marmaduke.woodman/miniconda3/envs/jlabdask/bin/python3.10 /home/marmaduke.woodman/miniconda3/envs/jlabdask/bin/dask-scheduler --host=10.11.7.58\n",
      "marmadu   409711  0.0  0.0  13056  3080 pts/15   Ss   15:07   0:00 /bin/bash -c ps ux | grep dask-scheduler\n",
      "marmadu   409713  0.0  0.0  12136  1064 pts/15   S    15:07   0:00 grep dask-scheduler\n"
     ]
    }
   ],
   "source": [
    "!ps ux | grep dask-scheduler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4acff2f-1958-48db-8ce5-0034994ede8a",
   "metadata": {},
   "source": [
    "Once the scheduler is running, the client is the interface,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "33250555-1114-4f99-b108-3590ec11a40a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\"> </div>\n",
       "    <div style=\"margin-left: 48px;\">\n",
       "        <h3 style=\"margin-bottom: 0px;\">Client</h3>\n",
       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Client-212c137b-5078-11ed-9640-dba408920301</p>\n",
       "        <table style=\"width: 100%; text-align: left;\">\n",
       "\n",
       "        <tr>\n",
       "        \n",
       "            <td style=\"text-align: left;\"><strong>Connection method:</strong> Direct</td>\n",
       "            <td style=\"text-align: left;\"></td>\n",
       "        \n",
       "        </tr>\n",
       "\n",
       "        \n",
       "            <tr>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Dashboard: </strong> <a href=\"http://10.11.7.58:8787/status\" target=\"_blank\">http://10.11.7.58:8787/status</a>\n",
       "                </td>\n",
       "                <td style=\"text-align: left;\"></td>\n",
       "            </tr>\n",
       "        \n",
       "\n",
       "        </table>\n",
       "\n",
       "        \n",
       "            <details>\n",
       "            <summary style=\"margin-bottom: 20px;\"><h3 style=\"display: inline;\">Scheduler Info</h3></summary>\n",
       "            <div style=\"\">\n",
       "    <div>\n",
       "        <div style=\"width: 24px; height: 24px; background-color: #FFF7E5; border: 3px solid #FF6132; border-radius: 5px; position: absolute;\"> </div>\n",
       "        <div style=\"margin-left: 48px;\">\n",
       "            <h3 style=\"margin-bottom: 0px;\">Scheduler</h3>\n",
       "            <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Scheduler-8eab6a79-8edd-4ba5-9b7e-e38643cabf63</p>\n",
       "            <table style=\"width: 100%; text-align: left;\">\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Comm:</strong> tcp://10.11.7.58:8786\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Workers:</strong> 0\n",
       "                    </td>\n",
       "                </tr>\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Dashboard:</strong> <a href=\"http://10.11.7.58:8787/status\" target=\"_blank\">http://10.11.7.58:8787/status</a>\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Total threads:</strong> 0\n",
       "                    </td>\n",
       "                </tr>\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Started:</strong> Just now\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Total memory:</strong> 0 B\n",
       "                    </td>\n",
       "                </tr>\n",
       "            </table>\n",
       "        </div>\n",
       "    </div>\n",
       "\n",
       "    <details style=\"margin-left: 48px;\">\n",
       "        <summary style=\"margin-bottom: 20px;\">\n",
       "            <h3 style=\"display: inline;\">Workers</h3>\n",
       "        </summary>\n",
       "\n",
       "        \n",
       "\n",
       "    </details>\n",
       "</div>\n",
       "            </details>\n",
       "        \n",
       "\n",
       "    </div>\n",
       "</div>"
      ],
      "text/plain": [
       "<Client: 'tcp://10.11.7.58:8786' processes=0 threads=0, memory=0 B>"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sched_url = 'tcp://10.11.7.58:8786'\n",
    "client = Client(sched_url)\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84925ff5-df2e-401d-9200-ae071586b8ad",
   "metadata": {},
   "source": [
    "Our scheduler doesn't have any workers yet, we schedule some via Slurm.  First have a look what's idle,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "ce2e4647-9ec8-455b-8427-69056d8127da",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "run*         up 7-00:00:00     14   idle c[1-4],m1,w[03,06,10-13,17-18,20] \n"
     ]
    }
   ],
   "source": [
    "!sinfo -p run | grep idle"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c309f88-cde4-41e7-a73e-09bddf09a137",
   "metadata": {},
   "source": [
    "Then start the workers, asking Slurm to ensure 4G memory available to each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "c8665c23-0548-4834-ba71-09c3c9778cd3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_workers = 16\n",
    "os.system(f'srun --mem-per-cpu 4G -p runcores -n {n_workers} dask-worker --nworkers 1 {sched_url} &> workers.log &')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "339ca7cc-4c82-40ec-9d57-0def45b6d2cd",
   "metadata": {},
   "source": [
    "Now we can check Slurm and `workers.log` to see workers are active.  If the number of workers requested is yet satisfied, wait a few seconds and re-evaluate the cell to check.  Or check `sinfo`, the nodes may not be available to satisfy that number of workers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "c03ddba3-c7d4-4bbf-973c-25765fb286c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON) \n",
      "           2722407  runcores dask-wor marmaduk  R       0:05      2 w[17-18] \n",
      "     1\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "     2\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "     3\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "     4\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "     5\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "     6\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "     7\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "     8\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "     9\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "    10\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "    11\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "    12\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "    13\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "    14\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "    15\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n",
      "    16\tdistributed.worker - INFO -         Registered to:      tcp://10.11.7.58:8786\n"
     ]
    }
   ],
   "source": [
    "!squeue -u $(whoami)\n",
    "!grep Register workers.log | nl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fb186e1-87ae-46e0-9f79-7695a526a2dc",
   "metadata": {},
   "source": [
    "and check Dask client sees the workers,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "5f6c47e9-7d96-49a0-9475-fe44d9444e75",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(client.scheduler_info()['workers'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "278da922-dffb-4607-9f0d-6e8a3a46630e",
   "metadata": {},
   "source": [
    "# Testing the Dask cluster\n",
    "\n",
    "Here we generate some random numbers and check overhead vs running locally.  The idea is simple so that we can see how this differs from regular Python functions.\n",
    "\n",
    "Regularly, generating a 16 by 16 matrix of normally distributed numbers would be just"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "92e787fb-5788-4401-9e54-fe36fe78b138",
   "metadata": {},
   "outputs": [],
   "source": [
    "z = np.random.randn(16, 16)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed0ff92c-692a-4f59-999e-ca11e4eb699d",
   "metadata": {},
   "source": [
    "This is also fast,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "73dfa35c-025d-4f17-a475-9cc92b4f4e0b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.62 µs ± 18 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit z = np.random.randn(16, 16)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbefa7f4-566c-4869-b5d8-cdac095ff7c1",
   "metadata": {},
   "source": [
    "With Dask we get \"future\"s which are a token for the result that will arrive, and we call the `result()` method to get the result. Following the example above, we ask each worker to generate one row of the 16 x 16 matrix,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "84d1979e-d947-46b8-8d92-912fda3bdd88",
   "metadata": {},
   "outputs": [],
   "source": [
    "futures = client.map(np.random.randn, [16 for _ in range(n_workers)])\n",
    "z = np.array([_.result() for _ in futures])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a67ad312-449e-4601-9db2-a1093d2730c8",
   "metadata": {},
   "source": [
    "In a small example we expect communication overhead to be significant,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "00100566-794f-44ca-8c30-b1c649fd8687",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dist_randn(m,n):\n",
    "    work_sizes = [n for _ in range(m)]\n",
    "    futures = client.map(np.random.randn, work_sizes)\n",
    "    return np.array([_.result() for _ in futures])\n",
    "\n",
    "%timeit dist_randn(16, 16)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f86eceb6-5e15-4620-b4a4-3072915f50b7",
   "metadata": {},
   "source": [
    "We can check that overhead,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "c2dfe03b-0684-4653-9d5d-c2051b806002",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.74 s ± 14 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit dist_randn(1024, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "3527f283-002f-42cc-961b-0e68105c8cfb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.00169921875"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1.74 / 1024"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "932ec217-9035-491f-86ff-afd5723ea5db",
   "metadata": {},
   "source": [
    "is about ~1.7ms per task, so for cases like parameter sweep simulations or neuroimaging analysis commands, this is insignificant. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48683d05-df86-450a-9ed5-aa3df2e4654c",
   "metadata": {},
   "source": [
    "# Using commands\n",
    "\n",
    "Running external non-Python programs should work as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "8fb534e2-a064-4515-9833-7a346b79f8c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 'w17'),\n",
       " (1, 'w17'),\n",
       " (2, 'w17'),\n",
       " (3, 'w17'),\n",
       " (4, 'w17'),\n",
       " (5, 'w17'),\n",
       " (6, 'w17'),\n",
       " (7, 'w18'),\n",
       " (8, 'w18'),\n",
       " (9, 'w18'),\n",
       " (10, 'w18'),\n",
       " (11, 'w18'),\n",
       " (12, 'w18'),\n",
       " (13, 'w18'),\n",
       " (14, 'w18'),\n",
       " (15, 'w17')]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import subprocess\n",
    "\n",
    "def get_hostname(i):\n",
    "    name = subprocess.check_output('hostname -s'.split()).decode('ascii').strip()\n",
    "    return i, name\n",
    "\n",
    "[_.result() for _ in client.map(get_hostname, range(16))]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6d627cd-6ed8-447b-b24b-86e87587ac73",
   "metadata": {},
   "source": [
    "So this generalizes to invokving FreeSurfer or some Singularity container that you've already prepared, e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "7792ed5e-a26c-49eb-af8e-c1629174ee08",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting dask_demo.def\n"
     ]
    }
   ],
   "source": [
    "%%file dask_demo.def\n",
    "BootStrap: library\n",
    "From: ubuntu\n",
    "OSVersion: jammy\n",
    "MirrorURL: http://us.archive.ubuntu.com/ubuntu/\n",
    "\n",
    "%environment\n",
    "    DEBIAN_FRONTEND=noninteractive\n",
    "\n",
    "%post\n",
    "    apt-get update\n",
    "    apt-get install -y stress-ng"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "7d3bb320-e2a9-41ce-b761-996f02d6d1b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "!singularity build -F --fakeroot dask_demo dask_demo.def &> dask_demo.log"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a5b3298-efe5-4031-8c21-676ab1019c25",
   "metadata": {},
   "source": [
    "For this demo, we just use the stress test program to run a CPU stressor for 10 seconds.  Testing that on the login node where Jupyter is running looks like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "7dbd9af6-75af-4757-ac20-ab4752ca5634",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stress-ng: info:  [409044] setting to a 10 second run per stressor\n",
      "stress-ng: info:  [409044] dispatching hogs: 1 cpu\n",
      "stress-ng: info:  [409044] successful run completed in 10.01s\n"
     ]
    }
   ],
   "source": [
    "!singularity exec dask_demo stress-ng -c 1 -t 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "67093991-74c9-44a4-8bd4-8c01422ec3df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "took 10.57655644416809 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[(0, 'stress-ng: info:  [21138] successful run completed in 10.00s'),\n",
       " (1, 'stress-ng: info:  [21065] successful run completed in 10.00s'),\n",
       " (2, 'stress-ng: info:  [21064] successful run completed in 10.00s'),\n",
       " (3, 'stress-ng: info:  [21070] successful run completed in 10.00s'),\n",
       " (4, 'stress-ng: info:  [21112] successful run completed in 10.01s'),\n",
       " (5, 'stress-ng: info:  [21140] successful run completed in 10.01s'),\n",
       " (6, 'stress-ng: info:  [21100] successful run completed in 10.00s'),\n",
       " (7, 'stress-ng: info:  [21072] successful run completed in 10.00s'),\n",
       " (8, 'stress-ng: info:  [16907] successful run completed in 10.00s'),\n",
       " (9, 'stress-ng: info:  [16945] successful run completed in 10.00s'),\n",
       " (10, 'stress-ng: info:  [16910] successful run completed in 10.00s'),\n",
       " (11, 'stress-ng: info:  [16961] successful run completed in 10.00s'),\n",
       " (12, 'stress-ng: info:  [16978] successful run completed in 10.00s'),\n",
       " (13, 'stress-ng: info:  [16947] successful run completed in 10.00s'),\n",
       " (14, 'stress-ng: info:  [16967] successful run completed in 10.00s'),\n",
       " (15, 'stress-ng: info:  [16904] successful run completed in 10.00s')]"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def run_stress(i):\n",
    "    cmd = 'singularity exec dask_demo stress-ng -c 1 -t 10'\n",
    "    out = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)\n",
    "    out = out.decode('ascii').strip()\n",
    "    lines = out.split('\\n')\n",
    "    return i, lines[-1]\n",
    "\n",
    "tic = time.time()\n",
    "results = [_.result() for _ in client.map(run_stress, range(16))]\n",
    "toc = time.time()\n",
    "print('took', toc - tic, 's')\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa7e137e-1c51-4859-bc25-836e503e0092",
   "metadata": {},
   "source": [
    "# Problems\n",
    "\n",
    "Ofc sometimes bad things happen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "c5917b2e-c794-4427-9c9b-18dd8eaf6f40",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\"> </div>\n",
       "    <div style=\"margin-left: 48px;\">\n",
       "        <h3 style=\"margin-bottom: 0px;\">Client</h3>\n",
       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Client-32e74176-506d-11ed-9640-dba408920301</p>\n",
       "        <table style=\"width: 100%; text-align: left;\">\n",
       "\n",
       "        <tr>\n",
       "        \n",
       "            <td style=\"text-align: left;\"><strong>Connection method:</strong> Direct</td>\n",
       "            <td style=\"text-align: left;\"></td>\n",
       "        \n",
       "        </tr>\n",
       "\n",
       "        \n",
       "\n",
       "        </table>\n",
       "\n",
       "        \n",
       "            <p>No scheduler connected.</p>\n",
       "        \n",
       "\n",
       "    </div>\n",
       "</div>"
      ],
      "text/plain": [
       "<Client: No scheduler connected>"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b71579ce-eb52-4bb2-870a-3ce3cba443fa",
   "metadata": {},
   "source": [
    "No scheduler.. look at the logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "af6b8712-85f8-4d9a-baf6-b15b98640072",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "distributed.core - INFO - Starting established connection\n",
      "distributed.scheduler - INFO - Register worker <WorkerState 'tcp://10.11.7.27:43463', status: undefined, memory: 0, processing: 0>\n",
      "distributed.scheduler - INFO - Starting worker compute stream, tcp://10.11.7.27:43463\n",
      "distributed.core - INFO - Starting established connection\n",
      "distributed.scheduler - INFO - Register worker <WorkerState 'tcp://10.11.7.27:43243', status: undefined, memory: 0, processing: 0>\n",
      "distributed.scheduler - INFO - Starting worker compute stream, tcp://10.11.7.27:43243\n",
      "distributed.core - INFO - Starting established connection\n",
      "distributed.scheduler - INFO - Register worker <WorkerState 'tcp://10.11.7.27:46291', status: undefined, memory: 0, processing: 0>\n",
      "distributed.scheduler - INFO - Starting worker compute stream, tcp://10.11.7.27:46291\n",
      "distributed.core - INFO - Starting established connection\n"
     ]
    }
   ],
   "source": [
    "!tail scheduler.log"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32826345-ae55-41a3-b344-4ce1efac29a2",
   "metadata": {},
   "source": [
    "If this happens, just restart the scheduler above.\n",
    "\n",
    "The workers might have timed out too, set a longer walltime for the srun command above for the workers. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}