In order to install the OpenSSL headers on Ubuntu, use
sudo apt -y install libssl-dev
In order to install the OpenSSL headers on Ubuntu, use
sudo apt -y install libssl-dev
To emulate a keypress to the Enter
key in Puppeteer, use
await page.keyboard.press("Enter");
The E
needs to be uppercase for this to work!
To emulate the user typing something on the keyboard, use
await page.keyboard.type("the text");
This will type the text extremely fast with virtually no delay between the characters.
In order to simulate the finite typing speed of real users, use
await page.keyboard.type("the text", {delay: 100});
instead. The delay betwen characters in this example is 100 Milliseconds, i.e. the emulated user types 10 characters per second.
In order to emulate a tab
key press in Puppeteer, use
await page.keyboard.press("Tab");
Full example:
// Minimal puppeteer example const puppeteer = require('puppeteer'); (async () => { const browser = await puppeteer.launch({defaultViewport: {width: 1920, height: 1080}}); const page = await browser.newPage(); await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'}); // Press tab 10 times (effectively scrolls down on techoverflow.net) for (let i = 0; i < 10; i++) { await page.keyboard.press("Tab"); } // Screenshot to verify result await page.screenshot({path: 'screenshot.png'}); // Cleanup await browser.close(); })();
You want to emulate a tab key
press using
await page.keyboard.press("tab");
but you get an error message like
(node:30594) UnhandledPromiseRejectionWarning: Error: Unknown key: "tab" at assert (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:270:11) at Keyboard._keyDescriptionForString (/home/uli/dev/myproject/node_modules/puppeteer/lib/Input.js:96:5) at Keyboard.down (/home/uli/dev/myproject/node_modules/puppeteer/lib/Input.js:44:30) at Keyboard.<anonymous> (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:112:23) at Keyboard.press (/home/uli/dev/myproject/node_modules/puppeteer/lib/Input.js:178:16) at Keyboard.<anonymous> (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:112:23) at /home/uli/dev/myproject/test.js:8:23 at processTicksAndRejections (internal/process/task_queues.js:85:5) (node:30594) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2) (node:30594) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
You need to use Tab
, not tab
! The T
needs to be uppercase!
Use
await page.keyboard.press("Tab");
Full example:
// Minimal puppeteer example const puppeteer = require('puppeteer'); (async () => { const browser = await puppeteer.launch({defaultViewport: {width: 1920, height: 1080}}); const page = await browser.newPage(); await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'}); // Press tab 10 times (effectively scrolls down on techoverflow.net) for (let i = 0; i < 10; i++) { await page.keyboard.press("Tab"); } // Screenshot to verify result await page.screenshot({path: 'screenshot.png'}); // Cleanup await browser.close(); })();
Use this snippet to set the value of an HTML <input>
element in Puppeteer:
const newInputValue = "test 123"; await page.evaluate(val => document.querySelector('.search-form-input').value = val, newInputValue);amp
Remember to replace '.search-form-input'
by whatever CSS selector is suitable to select your <input>
. Examples include 'input[name="username"]'
or '.username > input'
.
Full example:
// Minimal puppeteer example const puppeteer = require('puppeteer'); (async () => { const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'}); // Set input value const newInputValue = "test 123"; await page.evaluate(val => document.querySelector('.search-form-input').value = val, newInputValue); // Screenshot to verify result await page.screenshot({path: 'screenshot.png'}); // Cleanup await browser.close(); })();
Note that this method will work for any simple <input>
, however it might not work for some heavily Javascripted inputs which you can find on some modern websites.
You can take a screenshot in Puppeteer using
await page.screenshot({path: 'screenshot.png'});
The path is relative to the current working directory.
Want to have a screenshot in a size different to 800×600? See How to set screenshot size in Puppeteer?
Full example:
// Minimal puppeteer example const puppeteer = require('puppeteer'); (async () => { const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'}); // Screenshot to verify result await page.screenshot({path: 'screenshot.png'}); // Cleanup await browser.close(); })();
The easiest way to sleep using async/await
in NodeJS is the sleep-promise package:
npm install --save sleep-promise
Using that package you can simply use await sleep(milliseconds)
syntax like this:
const sleep = require('sleep-promise'); // In any async function: await sleep(2000); // Wait 2000 ms
Note that this sleep()
variant is fully asynchronous, IO and other asynchronous operations will still be able to continue in the background – sleep()
will not block the NodeJS process.
Full example:
const sleep = require('sleep-promise'); (async () => { console.log("This prints immediately"); await sleep(2000); console.log("This prints 2 seconds later"); })();
Install the sleep-promise package:
npm install --save sleep-promise
then you can use
const sleep = require('sleep-promise'); // In any async function: await sleep(2000); // Wait 2000 ms
Full example:
const sleep = require('sleep-promise'); (async () => { console.log("This prints immediately"); await sleep(2000); console.log("This prints 2 seconds later"); })();
Using Python (pyppeteer)? Check out Pyppeteer minimal network response interception example
This example shows you how to intercept network responses in puppeteer.
Note: This intercepts the response, not the request! This means you can’t abort the request before it is actually sent to the server, but you can read the content of the response! See Minimal puppeteer request interception example for an example on how to intercept requests.
const puppeteer = require('puppeteer'); (async () => { const browser = await puppeteer.launch(); const page = await browser.newPage(); // Enable response interception page.on('response', async (response) => { console.info("URL", response.request().url()); console.info("Method", response.request().method()) console.info("Response headers", response.headers()) console.info("Request headers", response.request().headers()) // Use this to get the content as text const responseText = await response.text(); // ... or as buffer (for binary data) const responseBuffer = await response.buffer(); // ... or as JSON, if it's a JSON (else, this will throw!) const responseObj = await response.json(); }) await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'}); // Make a screenshot await page.screenshot({path: 'screenshot.png'}); await browser.close(); })();
Using Python (pyppeteer)? Check out Pyppeteer minimal network request interception example
This example shows you how to intercept network requests in puppeteer:
Note: This intercepts the request, not the response! This means you can abort the request made, but you can’t read the content of the response! See Minimal puppeteer response interception example for an example on how to intercept responses.
const puppeteer = require('puppeteer'); (async () => { const browser = await puppeteer.launch(); const page = await browser.newPage(); // Enable request interception await page.setRequestInterception(true); page.on('request', async (request) => { console.info("URL", request.url()); console.info("Method", request.method()) console.info("Headers", request.headers()) return request.continue(); // Allow request to continue // return request.abort(); // use this instead to abort the request! }) await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'}); // Make a screenshot await page.screenshot({path: 'screenshot.png'}); await browser.close(); })();
The equivalent to this Python code which is using re.findall()
import re hashtag_regex = r"(\B#\w\w+)" hits = re.findall(hashtag_regex, "This is a string #with #hashtags") print(hits) # prints ['#with', '#hashtags']
in Javascript is
const string = "This is a string #with #hashtags"; const re = /(\B#\w\w+)/g; const hits = []; // Iterate hits let match = null; do { match = re.exec(string); if(match) { hits.push(match[0]); } } while (match); console.log(hits); // Prints [ '#with', '#hashtags' ]
You need to assign your regular expression to a variable like re
! If you use
match = /(\B#\w\w+)/g.exec(string); // WRONG ! Don't do this !
you will create an infinite loop which always generates the first hit in the string, if any!
You are trying to set the viewport size in Puppeteer using
const browser = await puppeteer.launch({ defaultViewport: '1920x1080' });
but you see an error message like
(node:15692) UnhandledPromiseRejectionWarning: Error: Protocol error (Emulation.setDeviceMetricsOverride): Invalid parameters width: integer value expected; height: integer value expected at /home/uli/dev/myproject/node_modules/puppeteer/lib/Connection.js:183:56 at new Promise (<anonymous>) at CDPSession.send (/home/uli/dev/myproject/node_modules/puppeteer/lib/Connection.js:182:12) at EmulationManager.emulateViewport (/home/uli/dev/myproject/node_modules/puppeteer/lib/EmulationManager.js:41:20) at Page.setViewport (/home/uli/dev/myproject/node_modules/puppeteer/lib/Page.js:808:54) at Page.<anonymous> (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:112:23) at Function.create (/home/uli/dev/myproject/node_modules/puppeteer/lib/Page.js:49:18) at processTicksAndRejections (internal/process/task_queues.js:85:5) at async Browser._createPageInContext (/home/uli/dev/myproject/node_modules/puppeteer/lib/Browser.js:177:18) at async /home/uli/dev/myproject/index.js:8:16 -- ASYNC -- at Target.<anonymous> (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:111:15) at Browser._createPageInContext (/home/uli/dev/myproject/node_modules/puppeteer/lib/Browser.js:177:31) at processTicksAndRejections (internal/process/task_queues.js:85:5) at async /home/uli/dev/myproject/index.js:8:16 -- ASYNC -- at Browser.<anonymous> (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:111:15) at /home/uli/dev/myproject/index.js:8:30 at processTicksAndRejections (internal/process/task_queues.js:85:5) (node:15692) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1) (node:15692) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
You need to use
{defaultViewport: {width: 1920, height: 1080}}
instead of
{defaultViewport: '1920x1080'}
Full example:
// Minimal puppeteer example const puppeteer = require('puppeteer'); (async () => { const browser = await puppeteer.launch({ defaultViewport: {width: 1920, height: 1080} }); const page = await browser.newPage(); await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'}); // Make a screenshot await page.screenshot({path: 'screenshot.png'}); await browser.close(); })();
You are trying to make a screenshot using Puppeteer like this:
await page.screenshot({path: 'screenshot.png'});
but the screenshot is always 800×600 pixels. How can you increase the size of the screenshot?
You can’t set the size for the screenshot, you need to set the size for the viewport in puppeteer.launch
:
const browser = await puppeteer.launch({ defaultViewport: {width: 1920, height: 1080} });
Full example to produce a 1920x1080
screenshot:
const puppeteer = require('puppeteer'); (async () => { const browser = await puppeteer.launch({ defaultViewport: {width: 1920, height: 1080} }); const page = await browser.newPage(); await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'}); // Wait until page has loaded completely // Make a screenshot await page.screenshot({path: 'screenshot.png'}); await browser.close(); })();
You can use this example as a starting point for your puppeteer application.
// Minimal puppeteer example const puppeteer = require('puppeteer'); (async () => { const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'}); // // YOUR CODE GOES HERE! // await browser.close(); })();
The screenshot is 800x600px by default and could look like this:
The pure Javascript equivalent to
$(window).load(function() { // Your code goes here! })
is
window.addEventListener("load", function() { // Your code goes here! });
Note that it’s not supported by IE8, but that should not be an issue in 2019.
To make strcmp case-insensitive, use strcasecmp from #include <strings.h>
. strcasecmp can be used in exactly the same way as strcmp.
To make strncmp case-insensitive, use strncasecmp from #include <strings.h>
. strncasecmp can be used in exactly the same way as strncmp.
Note that both of these will not deal with unicode characters correctly, but will work just fine in most applications.
This example shows you how to intercept and print the content of a JSON response requested via any AJAX request on a web page by using Pyppeteer:
import asyncio import json from pyppeteer import launch async def intercept_network_response(response): # In this example, we care only about responses returning JSONs if "application/json" in response.headers.get("content-type", ""): # Print some info about the responses print("URL:", response.url) print("Method:", response.request.method) print("Response headers:", response.headers) print("Request Headers:", response.request.headers) print("Response status:", response.status) # Print the content of the response try: # await response.json() returns the response as Python object print("Content: ", await response.json()) except json.decoder.JSONDecodeError: # NOTE: Use await response.text() if you want to get raw response text print("Failed to decode JSON from", await response.text()) async def main(): browser = await launch() page = await browser.newPage() page.on('response', intercept_network_response) await page.goto('https://instagram.com') await browser.close() asyncio.get_event_loop().run_until_complete(main())
Using Javascript (puppeteer)? Check out Minimal puppeteer response interception example
This example shows you how to intercept network responses in pyppeteer.
Note: This intercepts the response, not the request! This means you can abort the request before it is actually sent to the server, but you can’t read the content of the response! See Pyppetteer minimal network request interception example for an example on how to intercept requests.
import asyncio from pyppeteer import launch async def intercept_network_response(response): # In this example, we only care about HTML responses! if "text/html" in response.headers.get("content-type", ""): # Print some info about the responses print("URL:", response.url) print("Method:", response.request.method) print("Response headers:", response.headers) print("Request Headers:", response.request.headers) print("Response status:", response.status) # Print the content of the response print("Content: ", await response.text()) # NOTE: Use await response.json() if you want to get the JSON directly async def main(): browser = await launch() page = await browser.newPage() page.on('response', intercept_network_response) await page.goto('https://techoverflow.net') await browser.close() asyncio.get_event_loop().run_until_complete(main())
If you see this error message in Python:
NameError: name 'reponse' is not defined
you likely just mis-spelled response
!
reponse <-- your code, you're missing an 's'! response <-- correct