Python API

Images

exception RPA.Images.ImageNotFoundError

Bases: Exception

Raised when template matching fails.

args
with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

class RPA.Images.Images

Bases: object

Images is a library for general image manipulation. For image-based desktop automation, use the RPA.Desktop library.

Coordinates

The coordinates used in the library are pairs of x and y values that represent pixels. The upper left corner of the image or screen is (0, 0). The x-coordinate increases towards the right, and the y-coordinate increases towards the bottom.

Regions are represented as tuples of (left, top, right, bottom). For example, a 400 by 200-pixel region in the upper left corner would be (0, 0, 400, 200).

Template matching

Template matching refers to an operation where the (potential) location of a smaller image is searched from a larger image. It can be used for verifying certain conditions or locating UI elements for desktop or web automation.

Requirements

The default installation depends on Pillow library, which is used for general image manipulation operations.

For more robust and faster template matching, the library can use a combination of NumPy and OpenCV. They can be installed by opting in to the cv dependency:

pip install rpaframework[cv]

Examples

Robot Framework

The Images library can be imported and used directly in Robot Framework, for instance, for capturing screenshots or verifying something on the screen.

Desktop automation based on images should be done using the corresponding desktop library, i.e. RPA.Desktop.

*** Settings ***
Library    RPA.Images

*** Keywords ***
Should show success
    [Documentation]    Raises ImageNotFoundError if success image is not on screen
    Find template on screen    ${CURDIR}${/}success.png

Save screenshot to results
    [Documentation]    Saves screenshot of desktop with unique name
    ${timestamp}=      Get current date    result_format=%H%M%S
    Take screenshot    filename=${OUTPUT_DIR}${/}desktop_${timestamp}.png

Python

from RPA.Images import Images

def draw_matches_on_image(source, template):
    matches = lib.find_template_in_image(source, template)
    for match in matches:
        lib.show_region_in_image(source, match)

    source.save("matches.png")
ROBOT_LIBRARY_DOC_FORMAT = 'REST'
ROBOT_LIBRARY_SCOPE = 'GLOBAL'
crop_image(image, region, filename=None)

Crop an existing image.

Parameters

  • image – Image to crop

  • region – Region to crop image to

  • filename – Save cropped image to filename

find_template_in_image(image, template, region=None, limit=None, tolerance=None) → List[RPA.core.geometry.Region]

Attempt to find the template from the given image.

Parameters

  • image – Path to image or Image instance, used to search from

  • template – Path to image or Image instance, used to search with

  • limit – Limit returned results to maximum of limit.

  • region – Area to search from. Can speed up search significantly.

  • tolerance – Tolerance for matching, value between 0.1 and 1.0

Returns

List of matching regions

Raises

ImageNotFoundError – No match was found

get_pixel_color_in_image(image, point)

Get the RGB value of a pixel in the image.

Parameters

  • image – image to get pixel from

  • point – coordinates for pixel or Point object

show_region_in_image(image, region, color='red', width=5)

Draw a rectangle onto the image around the given region.

Parameters

  • image – image to draw onto

  • region – coordinates for region or Region object

  • color – color of rectangle

  • width – line width of rectangle

class RPA.Images.RGB(red: int, green: int, blue: int)

Bases: object

Container for a single RGB value.

blue: int = None
classmethod from_pixel(value)

Create RGB value from pillow getpixel() return value.

green: int = None
luminance()

Approximate (perceived) luminance for RGB value.

red: int = None
class RPA.Images.TemplateMatcher

Bases: object

Container class for different template matching methods.

DEFAULT_TOLERANCE = 0.95
LIMIT_FAILSAFE = 256
match(image, template, limit=None, tolerance=None)

Attempt to find the template in the given image.

Parameters

  • image – image to search from

  • template – image to search with

  • limit – maximum number of returned matches

  • tolerance – minimum correlation factor between template and image

Returns

list of regions that match criteria

property tolerance
RPA.Images.chunks(obj, size, start=0)

Convert obj container to list of chunks of size.

RPA.Images.clamp(minimum, value, maximum)

Clamp value between given minimum and maximum.

RPA.Images.to_image(obj)

Convert obj to instance of Pillow’s Image class.