Tutorial 2: Interacting with images in phenopype

In this tutorial we learn how to open and close images in phenopype, and how to use the interactive featues of the program. Phenopype uses OpenCV’s HighGUI module to display images and to allow users to interact with images. HighGUI has a few pros and cons:

+ native OpenCV GUI (no extra GUI libraries required)
+ the module is extremely fast in displaying images 
+ interactions (drawing and measuring) are possible 

- sometimes unstable (e.g. windows are not closed but freeze)
- displaying instructions is hacky (text is "painted" onto a displayed image) 
- user input (key strokes and mouse clicks) sometimes isn't captured properly

Currently Phenopype uses the standard HighGUI libraries that ship with the most recent precompiled opencv-contrib-python package, which is Qt for Linux and macOS, and Win32 UI on Windows. The Qt GUI is a bit more userfriendly with builtin buttons, scrollbars, RGB info and zoom, but you don’t actually need those things for basic Phenopype GUI interactions.

GUI control

IMPORTANT - read before continuing:

Open and close the GUI window

If you use the load_image function to load an image file into Python as an array, and then display it with show_image, a HighGUI window will pop up and display the array. While the window is open, the Python kernel is “busy”, and you cannot interact with the console, i.e. run any code - you first have to close the image. DON’T use the red closing button in the upper right - this will interrupt the connection to the kernel. Instead, do the following (make sure the window is highlighted):

  • Enter - close window or finish an interactive function in pype-mode

  • Ctrl+Enter - close and finish a window in pype-mode

  • Esc - close a window and quit the Phenoype process that invoked it. This may also work when the process is frozen.

Issues

  • If your keystore still isn’t recognized, make sure the window is highlighted (i.e. click on it) and try again.

  • If you killed a process but the window still open (i.e. the kernel is not busy anymore), type import cv2 and cv2.destroyAllWindows() into the console to close the window.

  • If a window and the Python kernel is frozen permanently, you need to restart it - sorry!

Opening images in phenopype

To open an image, it first has to be loaded as an array using load_image. The array can then be passed on to show_image, which simply displays an image (no interactions, except zooming in using the mousewheel). The window is closed by keystroke (Enter, or Esc which both closes the window and ends ongoing processes).

import phenopype as pp
import os

img = pp.load_image(os.path.join("data",'stickle1.jpg')) 
pp.show_image(img)

show_image can also handle multiple images. Here we loop through the images folder, attach all images to a list, and pass that list of arrays to the functions - it will give a warning if more than 10 images are being opened at the same time.

import os
images = [] # square-brakets make an empty list
names = os.listdir("data") # making a list of all the files names inside a directory

for i in names: # looping along our list of names
    filepath = os.path.join("data", i) # joining name and path strings 
    images.append(pp.load_image(filepath)) # load images and store them in list

pp.show_image(images, position_offset=100) ## show all images in the image folder in the tutorial directory
Invalid file extension ".mp4" - could not load image:

skipped showing list item of type NoneType
phenopype - 2
phenopype - 3
phenopype - 4
phenopype - 5
phenopype - 6

The function has a few more options to arrange and separte images across the screen. Future version of phenopype will do this in more meaningful manner (e.g., arrange small images side by side until the screen is filled).

pp.show_image(images, 
              window_max_dim=100,     # maximum dimension (in either direction) for the windows
              position_offset=100,    # window offset if multiple windows are displayed
              position_reset=True)    # reset window positions (i.e. window position will not be remembered from previous call)
skipped showing list item of type NoneType
phenopype - 2
phenopype - 3
phenopype - 4
phenopype - 5
phenopype - 6

Creating masks

Masking, i.e. removing unwanted parts of an image that contain noise by including or excluding certain parts of the image, is an important preprocessing step in any computer vision workflow. Phenopype’s create_mask tool provides flexibility when drawing masks.

Create masks

Fig. 1: Phenopype’s mask tool in action. You can include or exclude certain parts of the image; the resulting coordinates are recognized in subsequent computer vision steps (e.g. thresholding)

Using create_mask results in a DataFrame object that contains coordinates of the created mask. You can add multiple “submasks” that belong to the same mask layer that don’t have to be connected, and right click takes you one step back. Single polygons are finished with Ctrl, finish with Enter - the last open polyon will automatically be completed. Finish with Enter.

img = pp.load_image(os.path.join("data",'stickle1.jpg')) 
masks = pp.preprocessing.create_mask(img, tool="polygon", label="plates")
masks
poly
{'mask': {'a': {'info': {'annotation_type': 'mask',
    'phenopype_function': 'create_mask',
    'phenopype_version': '5.0.0'},
   'settings': {'tool': 'polygon',
    'line_width': 'auto',
    'line_colour': 'default',
    'label_size': 'auto',
    'label_width': 'auto',
    'label_colour': 'default'},
   'data': {'label': 'plates',
    'include': True,
    'n': 1,
    'mask': [[(1376, 264),
      (1382, 426),
      (1806, 428),
      (1812, 383),
      (1669, 345),
      (1489, 294),
      (1395, 271),
      (1376, 264)]]}}}}

We visualize the mask using draw_masks, which paints the coordinates onto a “canvas” is created with the original image and the mask coordinates, which can then be shown using show_image:

canvas = pp.visualization.select_canvas(img, canvas="raw")
canvas = pp.visualization.draw_mask(canvas, annotations=masks)    
pp.show_image(canvas)

We can add multiple masks to the same annotations file - in fact, you can add as many different annotations to the same annotations object as you want). For instance, try to drag a rectangle mask around the reference, and then draw it onto the same canvas. We need to give it a new annotation_id, otherwise it will override the default (“a”).

masks = pp.preprocessing.create_mask(img, annotation_id="b",  annotations=masks, tool="rectangle", label="scale-card") 
masks
{'mask': {'a': {'info': {'annotation_type': 'mask',
    'phenopype_function': 'create_mask',
    'phenopype_version': '5.0.0'},
   'settings': {'tool': 'polygon',
    'line_width': 'auto',
    'line_colour': 'default',
    'label_size': 'auto',
    'label_width': 'auto',
    'label_colour': 'default'},
   'data': {'label': 'plates',
    'include': True,
    'n': 1,
    'mask': [[(1376, 264),
      (1382, 426),
      (1806, 428),
      (1812, 383),
      (1669, 345),
      (1489, 294),
      (1395, 271),
      (1376, 264)]]}},
  'b': {'info': {'annotation_type': 'mask',
    'phenopype_function': 'create_mask',
    'phenopype_version': '5.0.0'},
   'settings': {'tool': 'rectangle',
    'line_width': 'auto',
    'line_colour': 'default',
    'label_size': 'auto',
    'label_width': 'auto',
    'label_colour': 'default'},
   'data': {'label': 'scale-card',
    'include': True,
    'n': 1,
    'mask': [[(266, 665),
      (1248, 665),
      (1248, 1554),
      (266, 1554),
      (266, 665)]]}}}}
canvas = pp.visualization.draw_mask(canvas, annotations=masks)    
pp.show_image(canvas)

With this tutorial we have learned how to handle images and some basic interactions - proceed with Tutorial 3 to get started with image anaysis and learn about the different workflows that phenopype offers.