{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial 2: Interacting with images in phenopype\n",
    "\n",
    "In this tutorial we learn how to open and close images in phenopype, and how to use the interactive featues of the program.  Phenopype uses OpenCV's HighGUI module to display images and to allow users to interact with images. HighGUI has a few pros and cons: \n",
    "\n",
    "```\n",
    "+ native OpenCV GUI (no extra GUI libraries required)\n",
    "+ the module is extremely fast in displaying images \n",
    "+ it can display very large image-arrays (> 10000x10000 pixels)\n",
    "+ it can display multiple images side by side\n",
    "+ interactions (drawing and measuring) are possible \n",
    "\n",
    "- sometimes unstable (e.g. windows are not closed but freeze)\n",
    "- issues with cross plattform stability (e.g. on macOS)\n",
    "- displaying instructions is hacky (text is \"painted\" onto a displayed image) \n",
    "- user input (key strokes and mouse clicks) sometimes isn't captured properly\n",
    "```\n",
    "\n",
    "Currently Phenopype uses the standard HighGUI libraries that ship with the most recent precompiled `opencv-contrib-python` package, which is `Qt` for Linux and macOS, and `Win32 UI` on Windows. The `Qt` GUI is a bit more userfriendly with builtin buttons, scrollbars, RGB info and zoom, but you don't actually need those things for basic Phenopype GUI interactions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info \">\n",
    "<div class=\"admonition seealso\">\n",
    "<p class=\"admonition-title\" > See also:</p>\n",
    "<p>\n",
    "\n",
    "*   [CV resources in the phenopype docs](https://www.phenopype.org/docs/resources/cv/)\n",
    "*   [OpenCV HighGUI docs](https://docs.opencv.org/master/dc/d46/group__highgui__qt.html)\n",
    "       \n",
    "</p>\n",
    "</div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GUI control\n",
    "\n",
    "<div class=\"alert alert-block alert-danger \">\n",
    "<div class=\"admonition attention\">\n",
    "<p class=\"admonition-title\">IMPORTANT - read before continuing:</p>\n",
    "    \n",
    "<p>\n",
    "\n",
    "**Open**\n",
    "\n",
    "In a very simple case, where you want to just inspect an image file from within phenopype, you use the `load_image` function to load an image file into Python as an array, and then display it with `show_image`. A HighGUI window will pop up and display the array. While the window is open, the Python kernel is \"busy\", and you cannot interact with the console, i.e. run any code - you first have to close the image again.\n",
    "\n",
    "**Close**\n",
    " \n",
    "Although the HighGUI window has the \"red crossed\" close button in the upper right corner, DO NOT USE IT! For practical reasons,  phenopype relies on key strokes to control the windows. Make sure that the window is selected / highlighted, and use the following key combinations to close it:  \n",
    "    \n",
    "- `Enter` - close window or finish an interactive function in `pype`-mode  \n",
    "- `Ctrl+Enter` - close and finish a window in `pype`-mode \n",
    "- `Esc` - close a window and quit the Phenoype process that invoked it. This may also work when the process is frozen. \n",
    "\n",
    "**Issues**\n",
    "\n",
    "Unfortunately the HighGUI isn't exactly stable, and may sometimes behave unexpectedly. Here are solutions to the most comment issues:\n",
    "\n",
    "- If a keystroke doen't do anything the first time, try a few times more (Phenopype \"listens\" to your keystroke while refreshing the presented image, and sometimes a refreshing operation overlaps with user input and it is not recognized).\n",
    "- If your keystore still isn't recognized, make sure the window is highlighted (i.e. click on it) and try again.\n",
    "- If you killed a process but the window still open (i.e. the kernel is not busy anymore), type \n",
    "`import cv2` and `cv2.destroyAllWindows()` into the console to close the window.\n",
    "- If a window and the Python kernel is frozen permanently, you need to restart it - sorry!\n",
    "    \n",
    "</p>\n",
    "</div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Opening images in phenopype\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To open an image, it first has to be loaded as an array using `load_image`. The array can then be passed on to `show_image`, which simply displays an image (no interactions, except zooming in using the mousewheel). The window is closed by keystroke (`Enter`, or `Esc` which both closes the window and ends ongoing processes). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import phenopype as pp\n",
    "import os\n",
    "\n",
    "img = pp.load_image(os.path.join(\"data\",'stickle1.jpg')) \n",
    "pp.show_image(img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`show_image` can also handle multiple images. Here we loop through the *images* folder, attach all images to a list, and pass that list of arrays to the functions - it will give a warning if more than 10 images are being opened at the same time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Invalid file extension \".mp4\" - could not load image:\n",
      "\n",
      "skipped showing list item of type NoneType\n",
      "phenopype - 2\n",
      "phenopype - 3\n",
      "phenopype - 4\n",
      "phenopype - 5\n",
      "phenopype - 6\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "images = [] # square-brakets make an empty list\n",
    "names = os.listdir(\"data\") # making a list of all the files names inside a directory\n",
    "\n",
    "for i in names: # looping along our list of names\n",
    "    filepath = os.path.join(\"data\", i) # joining name and path strings \n",
    "    images.append(pp.load_image(filepath)) # load images and store them in list\n",
    "\n",
    "pp.show_image(images, position_offset=100) ## show all images in the image folder in the tutorial directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function has a few more options to arrange and separte images across the screen. Future version of phenopype will do this in more meaningful manner (e.g., arrange small images side by side until the screen is filled).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "skipped showing list item of type NoneType\n",
      "phenopype - 2\n",
      "phenopype - 3\n",
      "phenopype - 4\n",
      "phenopype - 5\n",
      "phenopype - 6\n"
     ]
    }
   ],
   "source": [
    "pp.show_image(images, \n",
    "              window_max_dim=100,     # maximum dimension (in either direction) for the windows\n",
    "              position_offset=100,    # window offset if multiple windows are displayed\n",
    "              position_reset=True)    # reset window positions (i.e. window position will not be remembered from previous call)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating masks\n",
    "\n",
    "Masking, i.e. removing unwanted parts of an image that contain noise by including or excluding certain parts of the image, is an important preprocessing step in any computer vision workflow. Phenopype's `create_mask` tool provides flexibility when drawing masks.   \n",
    "\n",
    "<center>\n",
    "<div style=\"width:500px; text-align: left\" >\n",
    "    \n",
    "![Create masks](_figures/masks2.gif)\n",
    "    \n",
    "**Fig. 1:** Phenopype's mask tool in action. You can include or exclude certain parts of the image; the resulting coordinates are recognized in subsequent computer vision steps (e.g. thresholding)\n",
    "    \n",
    "</div>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `create_mask` results in a DataFrame object that contains coordinates of the created mask. You can add multiple \"submasks\" that belong to the same mask layer that don't have to be connected, and right click takes you one step back. Single polygons are finished with `Ctrl`, finish with `Enter` - the last open polyon will automatically be completed. Finish with `Enter`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mask': {'a': {'info': {'annotation_type': 'mask',\n",
       "    'phenopype_function': 'create_mask',\n",
       "    'phenopype_version': '3.2.3'},\n",
       "   'settings': {'tool': 'polygon',\n",
       "    'line_width': 5,\n",
       "    'line_colour': (0, 255, 0),\n",
       "    'label_size': 1,\n",
       "    'label_width': 1,\n",
       "    'label_colour': (0, 255, 0)},\n",
       "   'data': {'label': 'plates',\n",
       "    'include': True,\n",
       "    'n': 1,\n",
       "    'mask': [[(1373, 271),\n",
       "      (1376, 443),\n",
       "      (1859, 429),\n",
       "      (1878, 378),\n",
       "      (1702, 361),\n",
       "      (1390, 268),\n",
       "      (1373, 271)]]}}}}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "img = pp.load_image(os.path.join(\"data\",'stickle1.jpg')) \n",
    "mask_plates = pp.preprocessing.create_mask(img, tool=\"polygon\", label=\"plates\")\n",
    "mask_plates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating masks like this corresponds to the [low throughput workflow](tutorial_3.ipynb#Low-throughput-workflow), in which we first use `create_mask` to retrieve the mask coordinates, and then `draw_masks` to visualize the mask that was just created. A \"canvas\" is created with the original image and the mask coordinates, which can then be shown using `show_image`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "canvas = pp.visualization.draw_mask(img, \n",
    "                                    annotations=mask_plates, \n",
    "                                    label=True, \n",
    "                                    label_size=3, \n",
    "                                    line_width=2,\n",
    "                                    label_width=4)    \n",
    "pp.show_image(canvas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step, select the reference card to and set `include=False` to exclude it. We use the previously created canvas to know which areas have been masked already. Then we create and visualize a new canvas, which will draw both masks onto the original image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask_scale = pp.preprocessing.create_mask(\n",
    "    canvas, \n",
    "    label=\"scale\", \n",
    "    line_colour=\"red\",\n",
    "    include=False,\n",
    ")\n",
    "canvas = pp.visualization.draw_mask(\n",
    "    canvas, \n",
    "    annotations=mask_scale, \n",
    "    line_colour=\"red\",\n",
    "    label=True, \n",
    "    label_colour=\"red\",\n",
    "    label_size=3, \n",
    "    line_width=2,\n",
    "    label_width=4,\n",
    ")    \n",
    "pp.show_image(canvas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this tutorial we have learned how to handle images and some basic interactions - proceed with [Tutorial 3](tutorial_3.ipynb) to get started with image anaysis and learn about the different workflows that phenopype offers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}