Mouse on FlyOut Window on web application isn't working

Hello guys,

I’m currently testing RPA Express where I’m trying to automate a load into a web application.
Everything works fine (login, import button, file choice) till I arrive to a window which is a kind of ETL (column mapping, etc …). I don’t know why, but at this stage, the mouse isn’t working any more (click, move, ect…)… Any thought ?
NOTA : Keystrokes is working, but I would have to perform 44 tabs to arrive onto the “import” button :(:grimacing:!
Thanks in advance for your help.

(please check the log and the screenshot)

LOG

Error executing MouseClickAction
  com.workfusion.studio.rpa.recorder.playback.PlaybackException: Error executing TemplateAction[templateName=MouseClickAction.ftl,id=21,name=Optional[MouseClickAction],parent=1,nextSibling=22,arguments=ActionArguments[exactMatch=[true],imageName=[C:\\Users\\Boris\\workfusion-workspace\\rpae_project\\smartsheet\\1556303961352-anchor-1556303961372.apng],fullImageName=[1556303961352.png],xsi:type=[recorder:MouseClickAction, recorder:MouseClickAction],pollingInterval=[300],active=[true],searchInFrames=[false],type=[IMAGE],button=[1],delay=[0],timeoutMs=[10000],webElementLocatorType=[XPATH],x=[984],y=[664],awaitTimeout=[5000]]]
      at com.workfusion.studio.rpa.recorder.playback.flow.StandardControlFlow.execute(StandardControlFlow.java:54)
      at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.execute(TemplateAction.java:28)
      at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.execute(TemplateAction.java:15)
      at com.workfusion.studio.rpa.recorder.playback.player.ActionPlayer.next(ActionPlayer.java:81)
      at com.workfusion.studio.rpa.recorder.player.PlaybackLogic.playNextAction(PlaybackLogic.java:154)
      at com.workfusion.studio.rpa.recorder.player.PlaybackLogic.run(PlaybackLogic.java:112)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: org.openqa.selenium.WebDriverException: Image not found : 1556303961352-anchor-1556303961372.apng
Command duration or timeout: 0 milliseconds
Build info: version: '9.3.0.5', revision: '63553f3f1c', time: '2019-03-15T13:41:27.26Z'
System info: host: 'PC-BORIS', ip: '192.168.0.38', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_121'
Driver info: com.freedomoss.crowdcontrol.webharvest.selenium.wrapper.RemoteDriverWrapper
Capabilities [{imageSimilarityThreshold=0.8, extra.executor.id={Name=RPA Recorder}, CLOSE_ALL_WINDOWS=false, browserName=universal, javascriptEnabled=true, extra.capabilities.context={"browserType":"universal","startInPrivate":false,"blockImages":false,"maximizeOnStartup":false,"customCapabilities":{"platform":"WINDOWS","javascriptEnabled":true,"SEARCH_ALL_WINDOWS":true,"CLOSE_ALL_WINDOWS":false,"imageSimilarityThreshold":"0.8"},"executorId":{"Name":"RPA Recorder"}}, platformName=WINDOWS, SEARCH_ALL_WINDOWS=true, platform=WINDOWS}]
Session ID: c91dda04-3fd5-4383-af6c-634d52ab29ff
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:216)
      at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:168)
      at org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:41)
      at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:82)
      at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:45)
      at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:164)
      at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:741)
      at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:677)
      at com.workfusion.rpa.helpers.ImageElement.findImageCoordinates(ImageElement.java:156)
      at com.workfusion.rpa.helpers.ImageElement.getCoordinates(ImageElement.java:69)
      at com.workfusion.rpa.helpers.ImageElement.click(ImageElement.java:63)
      at com.workfusion.rpa.helpers.UiElement.click(UiElement.java:997)
      at org.openqa.selenium.WebElement$click.call(Unknown Source)
      at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
      at Script18.run(Script18.groovy:8)
      at com.workfusion.studio.rpa.recorder.playback.shell.GroovyShellWrapper.executeScript(GroovyShellWrapper.java:48)
      at com.workfusion.studio.rpa.recorder.playback.player.PlaybackContext.executeScript(PlaybackContext.java:108)
      at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.executeBehavior(TemplateAction.java:33)
      at com.workfusion.studio.rpa.recorder.playback.flow.StandardControlFlow.execute(StandardControlFlow.java:46)
      at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.execute(TemplateAction.java:28)
      at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.execute(TemplateAction.java:15)
      at com.workfusion.studio.rpa.recorder.playback.player.ActionPlayer.next(ActionPlayer.java:81)
      at com.workfusion.studio.rpa.recorder.player.PlaybackLogic.playNextAction(PlaybackLogic.java:154)
      at com.workfusion.studio.rpa.recorder.player.PlaybackLogic.run(PlaybackLogic.java:112)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: java.util.NoSuchElementException: Image not found : 1556303961352-anchor-1556303961372.apng
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
      at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77)
      at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
      at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:59)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:238)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:250)
      at Script1$1.call(Script1.groovy:12)
      at Script1$1.call(Script1.groovy)
      at com.workfusion.common.utils.SynchUtils.withFocusLock(SynchUtils.java:47)
      at com.workfusion.common.utils.SynchUtils$withFocusLock.call(Unknown Source)
      at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:136)
      at Script1.run(Script1.groovy:5)
      at groovy.lang.GroovyShell.evaluate(GroovyShell.java:444)
      at groovy.lang.GroovyShell.evaluate(GroovyShell.java:482)
      at groovy.lang.GroovyShell.evaluate(GroovyShell.java:453)
      at com.workfusion.autoit.driver.script.GroovyExecutor.execute(GroovyExecutor.java:45)
      at com.workfusion.autoit.driver.AutoItDriver.executeScriptInternal(AutoItDriver.java:235)
      at com.workfusion.autoit.driver.AutoItDriver.executeScript(AutoItDriver.java:190)
      at com.workfusion.universal.driver.UniversalDriver.executeScript(UniversalDriver.java:151)
      at org.openqa.selenium.remote.server.handler.ExecuteScript.call(ExecuteScript.java:54)
      at org.openqa.selenium.remote.server.handler.WebDriverHandler.handle(WebDriverHandler.java:41)
      at org.openqa.selenium.remote.server.rest.ResultConfig.handle(ResultConfig.java:134)
      at org.openqa.selenium.remote.server.JsonHttpCommandHandler.handleRequest(JsonHttpCommandHandler.java:206)
      at org.openqa.selenium.remote.server.InMemorySession.execute(InMemorySession.java:98)
      at org.openqa.selenium.remote.server.WebDriverServlet.lambda$handle$0(WebDriverServlet.java:231)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

Hi @BoDays according to the error text, the bot cannot find the image it is supposed to click on.
Are you sure the anchor image you use in the Mouse click action is present on the screen at this point?

Hi
@ashapkina,
Well, on the shared screenshot you should see the “import” image… But maybe, technically speaking it’s not an image… I don’t know.

NOTA : Yep, the “click on image” task is schedulded at the right moment (I even put a “wait” action before in order to be sure that the flyout window appear before it beggins).

If it’s not an image, would you have a proposition as a workaround ?
Thanks for your help.
Bodays

Try re-capturing the image for the mouse click action.

If it doesn’t work please share the screenshot of the mouse click action.

Hi Ashapkina,

  • I have tested with the “cancel” image (#1 on the screenshot) and it works perfectly.
  • I guess that it’s not working with the “import” image (#2 on the screenshot),
  • because it’s partially hidden by the small “action banner” (#3 on the screenshot)

How can I move this “action banner” in order to test and confirm (or not) my assumption ?

Thanks in advance.

Bodays

1 Like

Yes, this must be the reason.
There is no option to move the banner at the moment.

Try using only a part of the image of the Import button in the action, that is not covered by the banner.

1 Like

hanks Ashapkina, it has solvedT my issue :+1:t6:

2 Likes