Crawling Multiple URLs In A Loop Using Puppeteer

Answer :

map, forEach, reduce, etc, does not wait for the asynchronous operation within them, before they proceed to the next element of the iterator they are iterating over.

There are multiple ways of going through each item of an iterator synchronously while performing an asynchronous operation, but the easiest in this case I think would be to simply use a normal for operator, which does wait for the operation to finish.

const urls = [...]

for (let i = 0; i < urls.length; i++) {
    const url = urls[i];
    await page.goto(`${url}`);
    await page.waitForNavigation({ waitUntil: 'networkidle2' });
}

This would visit one url after another, as you are expecting. If you are curious about iterating serially using await/async, you can have a peek at this answer: https://stackoverflow.com/a/24586168/791691

The accepted answer shows how to serially visit each page one at a time. However, you may want to visit multiple pages simultaneously when the task is embarrassingly parallel, that is, scraping a particular page isn't dependent on data extracted from other pages.

A tool that can help achieve this is Promise.allSettled which lets us fire off a bunch of promises at once, determine which were successful and harvest results.

For a basic example, let's say we want to scrape usernames for Stack Overflow users given a series of ids.

Serial code:

const puppeteer = require("puppeteer");

(async () => {
  const browser = await puppeteer.launch({dumpio: false});
  const [page] = await browser.pages();
  const baseURL = "https://stackoverflow.com/users";
  const startId = 6243352;
  const qty = 5;
  const usernames = [];
  
  for (let i = startId; i < startId + qty; i++) {
    try {
      await page.goto(`${baseURL}/${i}`);
      usernames.push(await page.$eval(
        ".profile-user--name", 
        el => el.children[0].innerText
      ));
    }
    catch (err) {}
  }

  console.log(usernames.length);
  await browser.close();
})();

Parallel code:

const puppeteer = require("puppeteer");

(async () => {
  const browser = await puppeteer.launch({dumpio: false});
  const baseURL = "https://stackoverflow.com/users";
  const startId = 6243352;
  const qty = 5;

  const usernames = (await Promise.allSettled(
    [...Array(qty)].map(async (_, i) => {
      const page = await browser.newPage();
      await page.goto(`${baseURL}/${i + startId}`);
      return page.$eval(
        ".profile-user--name", 
        el => el.children[0].innerText
      );
    })))
    .filter(e => e.status === "fulfilled")
    .map(e => e.value)
  ;
  console.log(usernames.length);
  await browser.close();
})();

Remember that this is a technique, not a silver bullet that guarantees a speed increase on all workloads. It will take some experimentation to find the optimal balance between the cost of creating more pages versus the parallelization of network requests on a given particular task and system.

The example here is contrived since it's not interacting with the page dynamically, so there's not as much room for gain as in a typical Puppeteer use case that involves network requests and blocking waits per page.

Of course, beware of rate limiting and any other restrictions imposed by sites (running the code above may anger Stack Overflow's rate limiter).

For tasks where creating a page per task is prohibitively expensive or you'd like to set a cap on parallel request dispatches, consider using a task queue.

This pattern can also be extended to handle the case when certain pages depend on data from other pages, forming a dependency graph.

See also Using async/await with a forEach loop which explains why the original attempt in this thread using map fails to wait for each promise.