Image Recognition doesnt work anymore since updating to Lumen

image

#1

Hello RPAExpress-Team,

I just updated to RPA Express 2.0 Lumen. First thing I experienced was, that “Wait for Image” and “Click on Image” doesnt work anymore, no matter what image threshold I use. I also tried to replace the image by a new one and it just doesnt recognize the image.
What could be the reason for that?

Thank you and bests!


#2

HI Kevin,

where do you click on images, in a desktop application or in web?
Is it possible that it just doesn’t have time to load?


#3

Hi ashapkina,

thanks for your fast reply!
I click on it both, in a desktop application and in the web. And the page has loaded after 2-3 seconds, but the Wait for Image Action waits for a minute. So I guess both is not an explanation.

Bests!


#4

@khenrichs what does the error details say, Image not found or another error?


#5

There is no error thrown except for “failed to execute action” for the “Click on Image” case. For the “Wait for Image” case it just waits for the given amount of time and ends up not finding the image.

I guess it is some kind of migration problem:
We imported a script from rpa 1.x.x to Lumen. It doesnt work. But if we create the whole script from scratch in Lumen, it works (everything is identical).

And - as you would have guessed - starting all over writing all scripts from scratch is not a good solution :wink:


#6

@khenrichs could you copy the error details and post here?
Also, just to be sure: you play the recording from the Studio, and it is played on your local desktop, not a remote one, correct?


#7

Yes, its started from the Studio and played on my local desktop. Details:

Error executing MouseClickAction
com.workfusion.studio.rpa.recorder.playback.PlaybackException: Error executing TemplateAction[templateName=MouseClickAction.ftl,id=3,name=Optional[MouseClickAction],parent=-1,nextSibling=4,arguments=ActionArguments[exactMatch=[true],imageName=[1510052499158-anchor-1510052499171.apng],fullImageName=[1510052499158.png],xsi:type=[recorder:MouseClickAction, recorder:MouseClickAction],pollingInterval=[300],active=[true],searchInFrames=[false],type=[IMAGE],button=[1],offsetX=[173],delay=[5000],offsetY=[3],webElementLocatorType=[XPATH],actionDetails=[(click left button)],awaitTimeout=[4000]]]
at com.workfusion.studio.rpa.recorder.playback.flow.StandardControlFlow.execute(StandardControlFlow.java:54)
at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.execute(TemplateAction.java:30)
at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.execute(TemplateAction.java:17)
at com.workfusion.studio.rpa.recorder.playback.player.ActionPlayer.next(ActionPlayer.java:53)
at com.workfusion.studio.rpa.recorder.player.PlaybackLogic.playNextAction(PlaybackLogic.java:153)
at com.workfusion.studio.rpa.recorder.player.PlaybackLogic.run(PlaybackLogic.java:113)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.openqa.selenium.WebDriverException: Build info: version: ‘9.0.0.1’, revision: ‘e3a0fd7071’, time: ‘2018-05-11T11:35:20.018Z’
System info: host: ‘’, ip: ‘’, os.name: ‘Windows 10’, os.arch: ‘amd64’, os.version: ‘10.0’, java.version: ‘1.8.0_121’
Driver info: com.freedomoss.crowdcontrol.webharvest.selenium.wrapper.RemoteDriverWrapper
Capabilities [{imageSimilarityThreshold=0.8, extra.executor.id={Name=RPA Recorder}, CLOSE_ALL_WINDOWS=false, browserName=universal, javascriptEnabled=true, extra.capabilities.context={“browserType”:“universal”,“startInPrivate”:false,“blockImages”:false,“maximizeOnStartup”:false,“customCapabilities”:{“platform”:“WINDOWS”,“javascriptEnabled”:true,“SEARCH_ALL_WINDOWS”:true,“CLOSE_ALL_WINDOWS”:false,“imageSimilarityThreshold”:“0.8”},“executorId”:{“Name”:“RPA Recorder”}}, platformName=WINDOWS, SEARCH_ALL_WINDOWS=true, platform=WINDOWS}]
Session ID: ea67a04b-1136-4230-a073-bfa5f16d1d4a
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:216)
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:168)
at org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:41)
at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:82)
at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:45)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:164)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:656)
at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:592)
at com.workfusion.rpa.helpers.RPA.executeScript(RPA.java:864)
at com.workfusion.rpa.helpers.RPA.executeGroovyScript(RPA.java:930)
at com.workfusion.rpa.helpers.ImageElement.findImageCoordinates(ImageElement.java:156)
at com.workfusion.rpa.helpers.ImageElement.getCoordinates(ImageElement.java:69)
at com.workfusion.rpa.helpers.UiElement.getCoordinates(UiElement.java:1045)
at org.openqa.selenium.internal.Locatable$getCoordinates.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
at Script3.run(Script3.groovy:8)
at com.workfusion.studio.rpa.recorder.playback.shell.GroovyShellWrapper.executeScript(GroovyShellWrapper.java:90)
at com.workfusion.studio.rpa.recorder.playback.player.PlaybackContext.executeScript(PlaybackContext.java:65)
at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.executeBehavior(TemplateAction.java:35)
at com.workfusion.studio.rpa.recorder.playback.flow.StandardControlFlow.execute(StandardControlFlow.java:46)
at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.execute(TemplateAction.java:30)
at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.execute(TemplateAction.java:17)
at com.workfusion.studio.rpa.recorder.playback.player.ActionPlayer.next(ActionPlayer.java:53)
at com.workfusion.studio.rpa.recorder.player.PlaybackLogic.playNextAction(PlaybackLogic.java:153)
at com.workfusion.studio.rpa.recorder.player.PlaybackLogic.run(PlaybackLogic.java:113)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.sikuli.script.ImagePath.remove(ImagePath.java:459)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrap.invoke(StaticMetaMethodSite.java:133)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:91)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at Script1$1.call(Script1.groovy:16)
at Script1$1.call(Script1.groovy)
at com.workfusion.common.utils.SynchUtils.withFocusLock(SynchUtils.java:47)
at com.workfusion.common.utils.SynchUtils$withFocusLock.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
at Script1.run(Script1.groovy:5)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:585)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:623)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:594)
at com.workfusion.autoit.driver.AutoItDriver.executeGroovyScript(AutoItDriver.java:243)
at com.workfusion.autoit.driver.AutoItDriver.executeScript(AutoItDriver.java:211)
at com.workfusion.autoit.driver.AutoItDriver.executeScript(AutoItDriver.java:187)
at com.workfusion.universal.driver.UniversalDriver.executeScript(UniversalDriver.java:149)
at org.openqa.selenium.remote.server.handler.ExecuteScript.call(ExecuteScript.java:54)
at org.openqa.selenium.remote.server.handler.WebDriverHandler.handle(WebDriverHandler.java:41)
at org.openqa.selenium.remote.server.rest.ResultConfig.handle(ResultConfig.java:133)
at org.openqa.selenium.remote.server.JsonHttpCommandHandler.handleRequest(JsonHttpCommandHandler.java:203)
at org.openqa.selenium.remote.server.InMemorySession.execute(InMemorySession.java:98)
at org.openqa.selenium.remote.server.WebDriverServlet.lambda$handle$0(WebDriverServlet.java:231)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)


#8

@khenrichs This exception usually happens when the recording is played from the Studio on one of the remote desktops, not the local one. Have you tried playing it from Control Tower?
If it still doesn’t work, could you post the recording here so we have a look?


#9

Publishing to Control Tower doesnt work either. Maybe it takes some time until it appears in the list.

But I can’t post the recording, sorry! Thank you for your help, though :slight_smile:


#10

Could you send the screenshot (the whole screen) of the process after it errors out?
If you are afraid there might be some sensitive data, you can send it in a personal message.


#11

Alright, I just sent you a personal message :slight_smile: Thank you!


#12

Hi Kevin,

I just played the recording and it worked.

Did you create additional bots when you installed RPA Express 2.0? Perhaps, the recording is played on one of additional bots. Try this: open Platform Monitor, go to Control Pane - Bot Manager - localhost.
Stop additional bots (all except rpa.bot and rpa.bot_relay)

Try playing the recording again. Does it still give the same error?


#13

Hi Alesia,

thanks for testing so fast.
In the Bot Manager there are only two processes, rpa.bot and rpa.bot_relay. On the machine I work on I never had to decide if I wanna create additional bots.
I had to decide when I first installed it on another machine and created 9 bots. Does this affect an installation on another machine?


#14

Okay, got it. In which version was the recording originally created?
Was the backup file saved (like the ones on the screenshot)?
image


#15

It was created with 1.1.6.
The file was saved but I didnt send it to you since passwords were included :wink:


#16

On the machine I installed RPA Lumen first, I found a solution:
Importing the old script, exporting it right away and importing this exported script make it work again. But on the other machine that solution doesn’t seem to work…


#17

Hey again,
has anyone come across the same problem?
Alesia: are there any news about that?

Thank you :slight_smile:


#18

Hey there! I guess it’t time to push this again :wink:
So: we stil arent able to use older scripts with image recognition in Lumen 2.0.
This is pretty urgent… I just hope someone came across this too and found a solution.


#19

Hi Kevin,

Sorry for delay, but we’ve been testing this, and we have found two cases when this error might happen:

  1. When you play the script from the Studio, it is played on one of the additional WorkfusionRPABots that were created during the installation, so it doesn’t have access to the images on you local desktop.
  2. A user has created a new workspace, in which the projects are saved. In case the name of the workspace has spaces, special or non-latin characters etc., the bot might not be able to access the images in the project folder.

Is your case like any of these cases?


#20

Hi Alesia,

thank you so much, that solved the problem! We had _ in our path and now it works :slight_smile:

Have a great day!