Web Scraping with Request Promise in Google Cloud Functions



Request Promise is the standard library that is used in Node.js for web scraping and other web related requests, it's simple, compact, and easy to use.

Because Request Promise is so compact it uses much less memory than other libraries such as Puppeteer, and therefore is much faster to get a result.

How to Install Request Promise in Google Cloud Functions


  1. Navigate to cloud.google.com
  2. Launch the Google Cloud Shell from the top right, you may also read Launching the Google Cloud Shell Console
  3. Once the shell console loads type npm install -g npm to ensure npm is uptodate
  4. Next we need to install request, type npm install --save request
  5. Next to install request promise, type npm install --save request-promise
  6. Now we will install cheerio, a library to help fetch content from the page, to install cheerio type in npm install cheerio
  7. You should now have Request Promise, Cheerio, and the required dependencies installed, time to create your first project using my free templates to get you started.

Create a Node.js project with Request Promise


  1. Create a new Google Cloud function
  2. Copy the following code into the index.js and package.js files like follows:
  3. Code to copy embedded below, you can also find the code on my GitHub repository

  4. Click Create, your function may take a couple of minutes to create, you should get a green tick once it has finished.
  5. Click the URL to test that your code works, it should fetch the HTML code for the lukestoolkit blog and render it on screen, this will tell you whether the code is working.
  6. Now its your turn to start developing, review the documentation for Request Promise and Cheerio linked below.




Recommended Reading

Was this helpful?

Yes No


Comments