Anonymous Web Scraping using Chrome Browser Extension - Luminati

Hi all,

I did a search and I can’t seem to find anything related to this topic, but feel free to redirect me if I missed it.

We are trying to scrape some websites anonymously, and would like to install and enable the Luminati anonymization Chrome Extension in all of our browsing sessions. Can this be done via some custom script code?

Or has anyone else tried to use anonymization services such as Luminati and have you successfully used them in a WorkFusion script?

Thanks in advance!

Hi @Jim_Frost,

Try exporting your recording to a bot task and adding a custom capability to it as in this post Firefox does not save "Security Exception"

You will need to add a line options.addExtensions(new File("/path/to/extension.crx")) to the capability with the path to the extension you want to use.

Thanks Alesia!

This is not exactly what we were hoping for, but it is helpful. It points us in the right direction.

What we are going to try instead, is to insert a few script steps that will change the proxy settings in the Chrome browser once the script opens it, then at the end of the script, revert the proxy settings back to normal. In that we we can be sure we are using our anonimization proxy server every time.

Thanks again!!

Jim

You cannot do it in a custom script inside the recording, but you can add capabilities to use proxy in a bot task. Here are some examples:

<capability name="ie.usePerProcessProxy" value="true"/>
 
<!--capability name="proxy" value="{proxyType=DIRECT}"/-->
 
<!-- Choose the right proxy option for your needs. -->
 
<capability name="proxy">
    <script return="proxyValue"><![CDATA[
       proxyValue = new org.openqa.selenium.Proxy().setAutodetect(true);
    ]]></script>
</capability>
 
<capability name="proxy">
    <script return="proxyValue"><![CDATA[ 
       proxyValue = new org.openqa.selenium.Proxy().setProxyType(org.openqa.selenium.Proxy.ProxyType.SYSTEM);      
    ]]></script>
</capability>
 
<capability name="proxy">
    <script return="proxyValue"><![CDATA[
        proxyValue = new org.openqa.selenium.Proxy().setProxyAutoconfigUrl("proxyAutoconfigUrl");
    ]]></script>
</capability>```

 
<capability name="proxy">
    <script return="proxyValue"><![CDATA[
        proxyValue = new org.openqa.selenium.Proxy().setHttpProxy("httpProxy");
        proxyValue.setSocksUsername("userName").setSocksPassword("password");
    ]]></script>
</capability>

You can also find some details about using proxy in selenium here: https://www.seleniumhq.org/docs/04_webdriver_advanced.jsp

I hope it will be helpful

That’s great Alesia!

Could we also use keystrokes, and after opening Chrome, go to Settings, change the proxy, close the settings window, then continue?

Yes, sure, you can do that at the beginning of each session.

Thanks!! :+1:

1 Like