OCR action fails in RPA Express V2.0.5

Hello,

I seem to be having difficulty with the OCR function in RPA Express V2.0.5. I am doing the following:

  1. I am using TeamViewer to connect to an Android Device (Samsung S7) onto which I am installing a certain app.
  2. Now at some stage the Android App presents met with a key in the following format: xxxx-xxxx-xxxx-xxxx
  3. I need to copy that key somehow and put it in a variable to use at a later stage. My recording stops at the stage where I use the OCR function to capture only the key on this page, in the android app? I have created a variable (String) to which this key must be written to, because this key will differ every time.

Is there anything else I can do / can do better in order to capture this key in Android, that I need? I do not know why it stops/fails right there?

Please advise!

Regards.

Cobus Griesel

Hi @Cobus_Griesel

What error does it show when it fails? Can you share the error log?

That is the thing… it just stops with no error(s), even in the log file.

So what I did was this:

  1. I stopped the RPA service completely and closed RPA express.
  2. Then I cleared the log file (workspace-api) in C:\logs.
  3. The started RPA Express again and run my recording once again.
  4. This time it worked 100%!

So I don’t know if this is a bug or anything in the SW?

Regards.

@Cobus_Griesel not sure. We will try to investigate it. Please let us know if this issue occurs again.

@ashapkina, thanks I will do.

Thanks for your time! Much appreciated!

Regards,

@Cobus_Griesel

1 Like

Hi @ashapkina,

Upon further investigation it looks like the replay will stop on OCR if the OCR fails to recognize the characters, for instance if the section you are trying to read is too small or not recognizable.

I think this can be handled a bit better by RPA Express?

Regards,

@Cobus_Griesel

@Cobus_Griesel does it just stop or does it show this error: org.openqa.selenium.WebDriverException: Failed to execute script. null ?

Hello @ashapkina

It just stops. No error/warning is given - it just stops.

But I am not sure if my circumstance is special - I will explain why:

I have a Samsung S7 which I use for Testing. Now I use TeamViewer to connect / share the screen of the Samsung S7 on my PC screen. (I also run everything inside a VM PC).

So the data of the android app on my phone is basically syncing with the same Windows app on my VM PC. So when I install the app on my phone from the Google Play Store, it generates this “pairing key” which I need to copy via OCR recognition. But the text of this pairing key is quit small, so I have found that the OCR doesn’t work so well when the text to be recognized is too small.

So basically what I did to get it to work 99% of the time was to copy a whole section of recognized-OCR text into a variable, then extract only the 4x parts that I need to 4x other variables. Then I use the text functionality of RPA Express to check for capital letters (and when it finds a capital letter it changes it to a small letter) and then I write those 4x variables each to the correct text box in my Windows App.

Long story short… it works 99% of the time because sometimes it recognizes a “5” as an “S”. So then I have put all that inside a retry function which will retry for 3 times. After it still fails, it will exit and write to my report that it failed because of OCR and not because there is an actual bug…

I hope all that makes sense… wow!

Regards,

@Cobus_Griesel

1 Like

@Cobus_Griesel very good explanation. :+1: Thank you!
Yes, if the characters are very small, OCR might not recognize them.

I thought you didn’t use Exception Handling, and the recording just stopped without showing an error. But since you use exception handling, it makes sense.

Hi @ashapkina,

That is the thing… Even if I use error handling, the recording stops with the OCR step, without any error - it just stops.

So after I have then re-started the OCR service of RPA Express, it seems to be working fine. So in short… even if I start my OCR service with RPA Express, it seems that I have to restart it again before my OCR step works.

Def an issue on the RPA Express’s OCR side…

@Cobus_Griesel you are using a virtual machine, correct? In this case we are aware of this issue and working to solve it.

Hi, I’m getting the below error while executing OCR. Pls suggest.

Error executing OcrAction
com.workfusion.studio.rpa.recorder.playback.PlaybackException: Error executing TemplateAction[templateName=OcrAction.ftl,id=3,name=Optional[OcrAction],parent=-1,nextSibling=4,arguments=ActionArguments[varName=[email],imageName=[C:\Users\SB00465063\workfusion-workspace\rpae_project\OCR_Sample\1538394668568-anchor-1538394668619.apng],fullImageName=[1538394668568.png],xsi:type=[recorder:OcrAction, recorder:OcrAction],pollingInterval=[300],active=[true],type=[CONTROL],offsetX=[21],offsetY=[172],delay=[0],width=[345],actionDetails=[(to ‘email’ rectangle 345 x 30)],height=[30],awaitTimeout=[5000]]]
at com.workfusion.studio.rpa.recorder.playback.flow.StandardControlFlow.execute(StandardControlFlow.java:54)
at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.execute(TemplateAction.java:28)
at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.execute(TemplateAction.java:15)
at com.workfusion.studio.rpa.recorder.playback.player.ActionPlayer.next(ActionPlayer.java:53)
at com.workfusion.studio.rpa.recorder.player.PlaybackLogic.playNextAction(PlaybackLogic.java:152)
at com.workfusion.studio.rpa.recorder.player.PlaybackLogic.run(PlaybackLogic.java:112)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.openqa.selenium.WebDriverException: Failed to execute script. 500 null: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>Could not parse multipart servlet request; nested exception is java.io.IOException: The temporary upload location [C:\Users\SB00465063\AppData\Local\Temp\tomcat.3177244990060083393.15580\work\Tomcat\localhost\ROOT] is not valid
Build info: version: ‘9.0.0.4’, revision: ‘e6e278acd9’, time: ‘2018-06-19T14:40:11.651Z’
System info: host: ‘INPUHJPC07620’, ip: ‘10.10.24.5’, os.name: ‘Windows 10’, os.arch: ‘amd64’, os.version: ‘10.0’, java.version: ‘1.8.0_121’
Driver info: driver.version: UniversalDriver
Command duration or timeout: 0 milliseconds
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:216)
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:168)
at org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:41)
at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:82)
at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:45)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:164)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:656)
at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:592)
at com.workfusion.rpa.helpers.RPA.executeScript(RPA.java:864)
at com.workfusion.rpa.helpers.RPA.executeGroovyScript(RPA.java:930)
at com.workfusion.rpa.helpers.Ocr.proccessImage(Ocr.java:111)
at com.workfusion.rpa.helpers.Ocr.proccessImage(Ocr.java:35)
at com.workfusion.rpa.helpers.Ocr$proccessImage.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at Script3.run(Script3.groovy:6)
at com.workfusion.studio.rpa.recorder.playback.shell.GroovyShellWrapper.executeScript(GroovyShellWrapper.java:90)
at com.workfusion.studio.rpa.recorder.playback.player.PlaybackContext.executeScript(PlaybackContext.java:65)
at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.executeBehavior(TemplateAction.java:33)
at com.workfusion.studio.rpa.recorder.playback.flow.StandardControlFlow.execute(StandardControlFlow.java:46)
at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.execute(TemplateAction.java:28)
at com.workfusion.studio.rpa.recorder.playback.action.template.TemplateAction.execute(TemplateAction.java:15)
at com.workfusion.studio.rpa.recorder.playback.player.ActionPlayer.next(ActionPlayer.java:53)
at com.workfusion.studio.rpa.recorder.player.PlaybackLogic.playNextAction(PlaybackLogic.java:152)
at com.workfusion.studio.rpa.recorder.player.PlaybackLogic.run(PlaybackLogic.java:112)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.openqa.selenium.WebDriverException: Failed to execute script. 500 null: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>Could not parse multipart servlet request; nested exception is java.io.IOException: The temporary upload location [C:\Users\SB00465063\AppData\Local\Temp\tomcat.3177244990060083393.15580\work\Tomcat\localhost\ROOT] is not valid
Build info: version: ‘9.0.0.4’, revision: ‘e6e278acd9’, time: ‘2018-06-19T14:40:11.651Z’
System info: host: ‘INPUHJPC07620’, ip: ‘10.10.24.5’, os.name: ‘Windows 10’, os.arch: ‘amd64’, os.version: ‘10.0’, java.version: ‘1.8.0_121’
Driver info: driver.version: UniversalDriver
Build info: version: ‘9.0.0.4’, revision: ‘e6e278acd9’, time: ‘2018-06-19T14:40:11.651Z’
System info: host: ‘INPUHJPC07620’, ip: ‘10.10.24.5’, os.name: ‘Windows 10’, os.arch: ‘amd64’, os.version: ‘10.0’, java.version: ‘1.8.0_121’
Driver info: driver.version: UniversalDriver
at com.workfusion.autoit.driver.AutoItDriver.executeScript(AutoItDriver.java:185)
at com.workfusion.universal.driver.UniversalDriver.executeScript(UniversalDriver.java:149)
at org.openqa.selenium.remote.server.handler.ExecuteScript.call(ExecuteScript.java:54)
at org.openqa.selenium.remote.server.handler.WebDriverHandler.handle(WebDriverHandler.java:41)
at org.openqa.selenium.remote.server.rest.ResultConfig.handle(ResultConfig.java:133)
at org.openqa.selenium.remote.server.JsonHttpCommandHandler.handleRequest(JsonHttpCommandHandler.java:203)
at org.openqa.selenium.remote.server.InMemorySession.execute(InMemorySession.java:98)
at org.openqa.selenium.remote.server.WebDriverServlet.lambda$handle$0(WebDriverServlet.java:231)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

@sb00465063 try restarting RPA Express and you computer and running the script again. Let me know if it doesn’t help.

Hi,
That error happens because the field set for OCR is null. I used to encounter that error loads of times and not understand what was happening. Try making sure the OCR has some characters set to recognize in the area you ask it to search for, and it will give that random value, instead of pointing at a blank area. So, in simple words, you need to give it something to identify, instead of a simple black screen, otherwise it cries, saying failed to execute null.

1 Like

@f20150512a yes, we have such bug in the current build.

We have already fixed it, and the fix will be included in the next release of RPA Express.

Hi guys,

I’m finding that the OCR function fails every now and then for whatever reason I can’t find. I forgot to grab the error message, but I’m willing to update this post when the issue reoccurs as it always seems to.

A solution that works I’ve found is to stop and restart the OCR function on the RPA Express toolbar, but we feel very put off in doing this because this error happens unpredictably and always when we end up discovering the next day our processes that use OCR fail.

If anyone has experienced a similar behaviour and has found a solution or pattern that causes this problem, please let me know.

1 Like

Hi Martina,

It is caused by a bug we have in 2.0 and 2.1 versions.
We have fixed it for 2.2 version that we are expecting to release by the end of next week.

1 Like

Hi @martina_r @f20150512a @Cobus_Griesel

the version with this bug fix has been released