Click Action on Target Image Coordinates are Inaccurate

Action: Mouse Click
Target: Image
Version: 2.1.1
When relying on image recognition to click on certain elements (like on a remote desktop) this function works sufficiently. However, on some occasions, assigned XY coordinates become off by an inch or two which is just enough to miss the target element when replaying the recording. The anchors are clearly static on all cases, but while building a process, these coordinates tend to deviate enough to miss a step despite being never modified before and after this occurrence.

Any recommendations or advice would be appreciated.

Hi @martina_r could you share the details of the mouse click action and the screenshot of your pc screen at the moment the bot has to click the image?

Sure thing.
The following is a partial screenshot of the moment the button in question needs to be clicked with its surroundings; we need to click the “Close” button:

image

When it’s time to click “Close”, the cursor hits the “Help” button instead.
The next image is of the Click mouse component’s settings:

image

And here are the settings assigned to the image:

image

As a side note, it seems that including or excluding the button from the anchor region does not appear to solve the case. Neither is changing the click position to compensate the difference (the difference is too large) Additionally, this is the first time this business process stumbled at this step; it is one example of the many that occur.

@martina_r on the screen above, the Help button is located under the click position, so the bot clicks it. Try moving the Click position to the right. Or, if possible, you can use another anchor region. For example, the Close button itself

Hi @ashapkina
I’ve been playing around with the studio and as things turn out, it’s much more reliable to make the button itself the anchor region as you said. To additionally guarantee that the element is clicked, just capturing the button image alone guarantees accurate coordinates.

For example if I needed to click a “Print” button, just an image of this button suffices:

image

Thank you for your help! My business process is now working sufficiently.

1 Like

@martina_r glad I could help :slight_smile:

Hi @ashapkina
Sorry to bother, but it seems I’ve spoken too soon. The business process has had a couple good runs where it has done everything perfectly, but still manages to miss its targets every now and then. It’s not consistent enough for me to identify a pattern.

For example, if I wish to click on a specific checkbox based on the following image the process will be able to select it most of the time. Sometimes, it will select a checkbox next to it. Sometimes it misses the checkbox by a single pixel thus disabling it to perform the next action.

Referring to my previous example in an earlier post, pressing the “Print” button occasionally fumbles and somehow manages to select buttons just next to the Print button despite possessing no image data of those buttons.

I’ve tried making the required button the anchor region, and I’ve tried making other nearby static images anchor regions. Both approaches do not have a 99% accuracy rate. Is there any way around this other than excessive exception handlers?

Hi @martina_r If the bot clicks on the wrong checkbox, do the actions that come after that fail? If so, you can try putting them in the Try to complete block and if an exception occurs, navigate to this page/view again and click the checkbox again.

Could you also share the screenshot of the page on which you need to click in the Print button? We will add it to the examples for the image-based automation improvement

Sure thing, I can gladly contribute.

I have, in fact, resorted to the excessive use of Exception Handling blocks to get around the issue. Each click action leads to another screen to open up with a button or input label to be clicked on; naturally the following click actions will fail since the image we are looking for wouldn’t appear when it hits the wrong button.

However, because there are so many images involved with the process and there are so many different scenarios where a simple miss-click can fail (since the wrong coordinates it uses varies), it has become quite taxing to work around this issue.

I hope a solution/improvement surfaces soon :crossed_fingers:

1 Like