Bulk web data extraction automation

Dear Team,

I would like to know how to do bulk web data extraction automation.

The scenario is as follows:
I have a list of products on a list of webpages and their rating displayed, for each product having rating > 4 i have to click on the product and extract the details. Can someone guide me through.


Hi @kgunputh can you share the URL of the page on which you’ll need to look for the products?

Hi, @ashapkina, unfortunately the url is on a local domain but the concept is same as the link below:

Lets assume for each product i have to extract the rating of each product for all the webpages and compare the rating if (let’s say > 4), i have to open the product and extract the information.

Is it possible to do bulk web scrapping ?

Grateful if you could help me out.

Thank you @kgunputh

It looks like it is possible.

Once you open the page, you can:

  • save the ratings of all products on the page in a List using Xpath
  • save the link to each product’s page in a List using Xpath
  • for each saved rating in the List, if rating is:
    more than 4 - open the link of the corresponding product’s page (using index) and save the data using Xpath, increment the index by 1
    less than 4 - just increment the index by 1

Here is a similar sample script

data-scrape.rar (2.5 KB)

step 3 can fail sometimes as the website uses dynamic xpaths. In this case run the recording again.

Also, there is currently a bug in comparing decimal numbers, so you will need to multiply the rating by 100 to be able to compare it as a workaround.