{ "cells": [ { "cell_type": "markdown", "id": "e14673fa", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "# Supervised annotation in DeepOF - search for pre-established patterns" ] }, { "cell_type": "markdown", "id": "e3044fb9", "metadata": {}, "source": [ "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lucasmiranda42/deepof/blob/master/docs/source/tutorial_notebooks/deepof_supervised_tutorial.ipynb)" ] }, { "cell_type": "markdown", "id": "8a0170cc", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "##### What we'll cover:\n", " \n", "* Run the supervised annotation pipeline.\n", "* Generate Gantt plots with all traits.\n", "* Explore trait enrichment across conditions.\n", "* Visualize global embeddings using the retrieved traits." ] }, { "cell_type": "code", "execution_count": 1, "id": "764990b0", "metadata": {}, "outputs": [], "source": [ "# # If using Google colab, uncomment and run this cell and the one below to set up the environment\n", "# # Note: becasuse of how colab handles the installation of local packages, this cell will kill your runtime.\n", "# # This is not an error! Just continue with the cells below.\n", "# import os\n", "# !git clone -q https://github.com/mlfpm/deepof.git\n", "# !pip install -q -e deepof --progress-bar off\n", "# os.chdir(\"deepof\")\n", "# !curl --output tutorial_files.zip https://datashare.mpcdf.mpg.de/s/knF7t78isQuIAr0/download\n", "# !unzip tutorial_files.zip\n", "# os.kill(os.getpid(), 9)" ] }, { "cell_type": "code", "execution_count": 2, "id": "50799c68", "metadata": {}, "outputs": [], "source": [ "# os.chdir(\"deepof\")\n", "# import os, warnings\n", "# warnings.filterwarnings('ignore')" ] }, { "cell_type": "markdown", "id": "6048415d", "metadata": {}, "source": [ "Let's start by importing some packages. We'll use python's os library to handle paths, pickle to load saved objects, and the data entry API within DeepOF, located in deepof.data" ] }, { "cell_type": "code", "execution_count": 3, "id": "d64823a2", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "import os\n", "import pickle\n", "import deepof.data" ] }, { "cell_type": "markdown", "id": "ffdcefa1", "metadata": {}, "source": [ "We'll also need some plotting gear:" ] }, { "cell_type": "code", "execution_count": 4, "id": "e438d39f", "metadata": {}, "outputs": [], "source": [ "import deepof.visuals\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns" ] }, { "cell_type": "markdown", "id": "60b9158e", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "### Loading a previously initiated project" ] }, { "cell_type": "markdown", "id": "5d73cb96", "metadata": {}, "source": [ "As last time, we'll load the already created tutorial dataset for further processing:" ] }, { "cell_type": "code", "execution_count": 5, "id": "6382da27", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# Load a previously saved project\n", "my_deepof_project = deepof.data.load_project(\"./tutorial_files/tutorial_project\")" ] }, { "cell_type": "markdown", "id": "f47b171a", "metadata": {}, "source": [ "**NOTE** to better show how DeepOF deals with statistics, all results shown in the documentation version of this tutorial were obtained using the full SI dataset, containing a total of 53 animals. If you'd like to gain access to this dataset, check out the code availability statement of the main DeepOF paper." ] }, { "cell_type": "markdown", "id": "10776701", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "### Running the supervised annotation pipeline" ] }, { "cell_type": "markdown", "id": "2fdb66e9", "metadata": {}, "source": [ "The supervised pipeline within DeepOF aims at providing simple but generalizable annotators for a variety of single and dyadic traits (as depicted in the figure below). Each of these traits is tagged independently, which means that more than one label can be applied per time point!" ] }, { "cell_type": "markdown", "id": "3c20a0d5", "metadata": {}, "source": [ "![supervised](./Assets/deepof_supervised.png)" ] }, { "cell_type": "markdown", "id": "b6c3b1c9", "metadata": {}, "source": [ "Running the pipeline is as simple as executing the `.supervised_annotation()` method in your Coordinates object. A progress bar will indicate how many videos have been processed already. Let's run it and see what happens:" ] }, { "cell_type": "code", "execution_count": 6, "id": "d368119b", "metadata": { "pycharm": { "name": "#%%\n" }, "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:16<00:00, 2.74s/it]\n" ] } ], "source": [ "supervised_annotation = my_deepof_project.supervised_annotation()" ] }, { "cell_type": "markdown", "id": "7cc204fc", "metadata": {}, "source": [ "It worked! Upon inspection, we see that supervised annotations contains a dictionary-like object with experiment IDs as keys, and data frames with annotations as values:" ] }, { "cell_type": "code", "execution_count": 7, "id": "d49995fe", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | B_W_nose2nose | \n", "B_W_sidebyside | \n", "B_W_sidereside | \n", "B_W_nose2tail | \n", "W_B_nose2tail | \n", "B_W_nose2body | \n", "W_B_nose2body | \n", "B_W_following | \n", "W_B_following | \n", "B_climbing | \n", "... | \n", "B_huddle | \n", "B_lookaround | \n", "B_speed | \n", "W_climbing | \n", "W_sniffing | \n", "W_huddle | \n", "W_lookaround | \n", "W_speed | \n", "B_missing | \n", "W_missing | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
00:00:00 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "1.0 | \n", "0.0 | \n", "0.000 | \n", "0.0 | \n", "0.0 | \n", "1.0 | \n", "0.0 | \n", "0.000 | \n", "0 | \n", "0 | \n", "
00:00:00.039935995 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "1.0 | \n", "0.0 | \n", "0.000 | \n", "0.0 | \n", "0.0 | \n", "1.0 | \n", "0.0 | \n", "0.000 | \n", "0 | \n", "0 | \n", "
00:00:00.079871991 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "1.0 | \n", "0.0 | \n", "0.000 | \n", "0.0 | \n", "0.0 | \n", "1.0 | \n", "0.0 | \n", "0.000 | \n", "0 | \n", "0 | \n", "
00:00:00.119807987 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "0.000 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.000 | \n", "0 | \n", "0 | \n", "
00:00:00.159743982 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "0.0 | \n", "1.0 | \n", "2.236 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "10.636 | \n", "0 | \n", "0 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
00:09:58.800320021 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "6.593 | \n", "0.0 | \n", "0.0 | \n", "1.0 | \n", "1.0 | \n", "0.086 | \n", "0 | \n", "0 | \n", "
00:09:58.840256017 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "4.934 | \n", "0.0 | \n", "0.0 | \n", "1.0 | \n", "1.0 | \n", "0.083 | \n", "0 | \n", "0 | \n", "
00:09:58.880192012 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "5.557 | \n", "0.0 | \n", "0.0 | \n", "1.0 | \n", "1.0 | \n", "0.064 | \n", "0 | \n", "0 | \n", "
00:09:58.920128008 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "6.130 | \n", "0.0 | \n", "0.0 | \n", "1.0 | \n", "1.0 | \n", "0.046 | \n", "0 | \n", "0 | \n", "
00:09:58.960064004 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "... | \n", "0.0 | \n", "0.0 | \n", "4.628 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "1.0 | \n", "0.031 | \n", "0 | \n", "0 | \n", "
14999 rows × 21 columns
\n", "