Technologies

How to emulate the Enter key in Puppeteer

To emulate a keypress to the Enter key in Puppeteer, use

await page.keyboard.press("Enter");

The E needs to be uppercase for this to work!

Posted by Uli Köhler in Javascript, Puppeteer

How to emulate keyboard input in Puppeteer

To emulate the user typing something on the keyboard, use

await page.keyboard.type("the text");

This will type the text extremely fast with virtually no delay between the characters.

In order to simulate the finite typing speed of real users, use

await page.keyboard.type("the text", {delay: 100});

instead. The delay betwen characters in this example is 100 Milliseconds, i.e. the emulated user types 10 characters per second.

Posted by Uli Köhler in Javascript, NodeJS, Puppeteer

How to emulate TAB key press in Puppeteer

In order to emulate a tab key press in Puppeteer, use

await page.keyboard.press("Tab");

Full example:

// Minimal puppeteer example
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch({defaultViewport: {width: 1920, height: 1080}});
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Press tab 10 times (effectively scrolls down on techoverflow.net)
  for (let i = 0; i < 10; i++) {
      await page.keyboard.press("Tab");
  }
  // Screenshot to verify result
  await page.screenshot({path: 'screenshot.png'});
  // Cleanup
  await browser.close();
})();

 

Posted by Uli Köhler in Javascript, Puppeteer

How to set <input> value in Puppeteer

Use this snippet to set the value of an HTML <input> element in Puppeteer:

const newInputValue = "test 123";
await page.evaluate(val => document.querySelector('.search-form-input').value = val, newInputValue);amp

Remember to replace '.search-form-input' by whatever CSS selector is suitable to select your <input>. Examples include 'input[name="username"]' or '.username > input'.

Full example:

// Minimal puppeteer example
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Set input value
  const newInputValue = "test 123";
  await page.evaluate(val => document.querySelector('.search-form-input').value = val, newInputValue);
  // Screenshot to verify result
  await page.screenshot({path: 'screenshot.png'});
  // Cleanup
  await browser.close();
})();

Note that this method will work for any simple <input>, however it might not work for some heavily Javascripted inputs which you can find on some modern websites.

Posted by Uli Köhler in Javascript, Puppeteer

How to save screenshot in Puppeteer as PNG?

You can take a screenshot in Puppeteer using

await page.screenshot({path: 'screenshot.png'});

The path is relative to the current working directory.

Want to have a screenshot in a size different to 800×600? See How to set screenshot size in Puppeteer?

Full example:

// Minimal puppeteer example
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Screenshot to verify result
  await page.screenshot({path: 'screenshot.png'});
  // Cleanup
  await browser.close();
})();

 

Posted by Uli Köhler in Javascript, NodeJS, Puppeteer

Minimal puppeteer response interception example

Using Python (pyppeteer)? Check out Pyppetteer minimal network response interception example

This example shows you how to intercept network responses in puppeteer.

Note: This intercepts the response, not the request! This means you can abort the request before it is actually sent to the server, but you can’t read the content of the response! See Minimal puppeteer request interception example for an example on how to intercept requests.

const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  // Enable response interception
  page.on('response', async (response) => {
      console.info("URL", response.request().url());
      console.info("Method", response.request().method())
      console.info("Response headers", response.headers())
      console.info("Request headers", response.request().headers())
      // Use this to get the content as text
      const responseText = await response.text();
      // ... or as buffer (for binary data)
      const responseBuffer = await response.buffer();
      // ... or as JSON, if it's a JSON (else, this will throw!)
      const responseObj = await response.json();
  })
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Make a screenshot
  await page.screenshot({path: 'screenshot.png'});
  await browser.close();
})();

 

Posted by Uli Köhler in Javascript, Puppeteer

Minimal puppeteer request interception example

Using Python (pyppeteer)? Check out Pyppetteer minimal network request interception example

This example shows you how to intercept network requests in puppeteer:

Note: This intercepts the request, not the response! This means you can abort the request made, but you can’t read the content of the response! See Minimal puppeteer response interception example for an example on how to intercept responses.

const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  // Enable request interception
  await page.setRequestInterception(true);
  page.on('request', async (request) => {
      console.info("URL", request.url());
      console.info("Method", request.method())
      console.info("Headers", request.headers())
      return request.continue(); // Allow request to continue
      // return request.abort(); // use this instead to abort the request!
  })
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Make a screenshot
  await page.screenshot({path: 'screenshot.png'});
  await browser.close();
})();

 

Posted by Uli Köhler in Javascript, Puppeteer

How to fix Puppeteer ‘Error: Protocol error (Emulation.setDeviceMetricsOverride): Invalid parameters width: integer value expected; height: integer value expected’

Problem:

You are trying to set the viewport size in Puppeteer using

const browser = await puppeteer.launch({
    defaultViewport: '1920x1080'
});

but you see an error message like

(node:15692) UnhandledPromiseRejectionWarning: Error: Protocol error (Emulation.setDeviceMetricsOverride): Invalid parameters width: integer value expected; height: integer value expected
    at /home/uli/dev/myproject/node_modules/puppeteer/lib/Connection.js:183:56
    at new Promise (<anonymous>)
    at CDPSession.send (/home/uli/dev/myproject/node_modules/puppeteer/lib/Connection.js:182:12)
    at EmulationManager.emulateViewport (/home/uli/dev/myproject/node_modules/puppeteer/lib/EmulationManager.js:41:20)
    at Page.setViewport (/home/uli/dev/myproject/node_modules/puppeteer/lib/Page.js:808:54)
    at Page.<anonymous> (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:112:23)
    at Function.create (/home/uli/dev/myproject/node_modules/puppeteer/lib/Page.js:49:18)
    at processTicksAndRejections (internal/process/task_queues.js:85:5)
    at async Browser._createPageInContext (/home/uli/dev/myproject/node_modules/puppeteer/lib/Browser.js:177:18)
    at async /home/uli/dev/myproject/index.js:8:16
  -- ASYNC --
    at Target.<anonymous> (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:111:15)
    at Browser._createPageInContext (/home/uli/dev/myproject/node_modules/puppeteer/lib/Browser.js:177:31)
    at processTicksAndRejections (internal/process/task_queues.js:85:5)
    at async /home/uli/dev/myproject/index.js:8:16
  -- ASYNC --
    at Browser.<anonymous> (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:111:15)
    at /home/uli/dev/myproject/index.js:8:30
    at processTicksAndRejections (internal/process/task_queues.js:85:5)
(node:15692) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:15692) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Solution:

You need to use

{defaultViewport: {width: 1920, height: 1080}}

instead of

{defaultViewport: '1920x1080'}

Full example:

// Minimal puppeteer example
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch({
      defaultViewport: {width: 1920, height: 1080}
  });
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Make a screenshot
  await page.screenshot({path: 'screenshot.png'});
  await browser.close();
})();

 

Posted by Uli Köhler in Javascript, Puppeteer

How to set screenshot size in Puppeteer?

Problem:

You are trying to make a screenshot using Puppeteer like this:

await page.screenshot({path: 'screenshot.png'});

but the screenshot is always 800×600 pixels. How can you increase the size of the screenshot?

Solution:

You can’t set the size for the screenshot, you need to set the size for the viewport in puppeteer.launch:

const browser = await puppeteer.launch({
    defaultViewport: {width: 1920, height: 1080}
});

Full example to produce a 1920x1080 screenshot:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
      defaultViewport: {width: 1920, height: 1080}
  });
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Wait until page has loaded completely
  // Make a screenshot
  await page.screenshot({path: 'screenshot.png'});

  await browser.close();
})();

 

Posted by Uli Köhler in Javascript, Puppeteer

Puppeteer minimal example

You can use this example as a starting point for your puppeteer application.

// Minimal puppeteer example
const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  //
  // YOUR CODE GOES HERE!
  //
  await browser.close();
})();

The screenshot is 800x600px by default and could look like this:

Posted by Uli Köhler in Javascript, NodeJS, Puppeteer

How to intercept AJAX JSON response in Pyppeteer

This example shows you how to intercept and print the content of a JSON response requested via any AJAX request on a web page by using Pyppeteer:

import asyncio
import json
from pyppeteer import launch

async def intercept_network_response(response):
    # In this example, we care only about responses returning JSONs
    if "application/json" in response.headers.get("content-type", ""):
        # Print some info about the responses
        print("URL:", response.url)
        print("Method:", response.request.method)
        print("Response headers:", response.headers)
        print("Request Headers:", response.request.headers)
        print("Response status:", response.status)
        # Print the content of the response
        try:
            # await response.json() returns the response as Python object
            print("Content: ", await response.json())
        except json.decoder.JSONDecodeError:
            # NOTE: Use await response.text() if you want to get raw response text
            print("Failed to decode JSON from", await response.text())

async def main():
    browser = await launch()
    page = await browser.newPage()
    
    page.on('response', intercept_network_response)
            
    await page.goto('https://instagram.com')
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())
Posted by Uli Köhler in Puppeteer, Python

Pyppetteer minimal network response interception example

Using Javascript (puppeteer)? Check out Minimal puppeteer response interception example

This example shows you how to intercept network responses in pyppeteer.

Note: This intercepts the response, not the request! This means you can abort the request before it is actually sent to the server, but you can’t read the content of the response! See Pyppetteer minimal network request interception example for an example on how to intercept requests.

import asyncio
from pyppeteer import launch

async def intercept_network_response(response):
    # In this example, we only care about HTML responses!
    if "text/html" in response.headers.get("content-type", ""):
        # Print some info about the responses
        print("URL:", response.url)
        print("Method:", response.request.method)
        print("Response headers:", response.headers)
        print("Request Headers:", response.request.headers)
        print("Response status:", response.status)
        # Print the content of the response
        print("Content: ", await response.text())
        # NOTE: Use await response.json() if you want to get the JSON directly

async def main():
    browser = await launch()
    page = await browser.newPage()
    
    page.on('response', intercept_network_response)
            
    await page.goto('https://techoverflow.net')
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

 

Posted by Uli Köhler in Puppeteer, Python

Pyppetteer minimal network request interception example

Using Javascript (puppeteer)? Check out Minimal puppeteer request interception example

This example shows you how to intercept network requests in pyppeteer:

Note: This intercepts the request, not the response! This means you can abort the request made, but you can’t read the content of the response! See Pyppetteer minimal network response interception example for an example on how to intercept responses.

import asyncio
from pyppeteer import launch

async def intercept_network_request(request):
    # Print some info about the request
    print("URL:", request.url)
    print("Method:", request.method)
    print("Headers:", request.headers)
    # NOTE: You can also await request.abort() to abort the requst1
    await request.continue_()

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.setRequestInterception(True)
    
    page.on('request', intercept_network_request)
            
    await page.goto('https://techoverflow.net')
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

 

Posted by Uli Köhler in Puppeteer, Python

How to fix pyppeteer.errors.NetworkError: Request interception is not enabled.

Note: Also see Pyppetteer minimal network request interception example

Problem:

You are trying to intercept a request in Pyppeteer using

page.on('request', my_intercept_request)

but you’re getting an error message like this:

Traceback (most recent call last):
  File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/dist-packages/pyee/_compat.py", line 62, in _callback
    self.emit('error', exc)
  File "/usr/local/lib/python3.6/dist-packages/pyee/_base.py", line 106, in emit
    self._emit_handle_potential_error(event, args[0] if args else None)
  File "/usr/local/lib/python3.6/dist-packages/pyee/_base.py", line 83, in _emit_handle_potential_error
    raise error
  File "run.py", line 6, in intercept
    await request.continue_()
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/network_manager.py", line 481, in continue_
    raise NetworkError('Request interception is not enabled.')
pyppeteer.errors.NetworkError: Request interception is not enabled.

Solution:

Add

await page.setRequestInterception(True)

directly after your call to

page = await browser.newPage()

This will enable request interception and your code will run just fine.

Posted by Uli Köhler in Puppeteer, Python

How to capture Raspi Camera image using OpenCV & Python

First, install OpenCV for Python 3:

sudo apt install python3-opencv

Here’s the code to acquire the image and store it in image.png:

#!/usr/bin/env python3
import cv2
video_capture = cv2.VideoCapture(0)
# Check success
if not video_capture.isOpened():
    raise Exception("Could not open video device")
# Read picture. ret === True on success
ret, frame = video_capture.read()

cv2.imwrite('image.png', frame)
# Close device
video_capture.release()

Run it using

python3 cv-raspicapture.py

 

Posted by Uli Köhler in OpenCV, Python, Raspberry Pi

How to fix MongoDB ‘exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /var/lib/mongodb, terminating’

Problem:

When trying to start MongoDB e.g. using sudo systemctl start mongod, the process terminates with status code 100 and you see this error message in /var/log/mongodb/mongodb.log:

2019-07-22T17:11:07.858+0200 I STORAGE  [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /var/lib/mongodb, terminating

Solution:

Fix the permissions of /var/lib/mongodb:

sudo chown -R mongodb: /var/lib/mongodb

then restart MongoDB e.g. using

sudo systemctl restart mongod

In case that does not help, check your error message if you are using a data directory different to /var/lib/mongodb. In that case run

sudo chown -R mongodb: <insert your data directory here>

In case that is not the case or doesn’t help either, check if your filesystem is mounted in read-only mode.

Posted by Uli Köhler in MongoDB

Installing apt packages using cloud-init

This cloud-init example installs nginx on Debian- or Ubuntu- based systems:

packages:
- nginx

If you want to enable upgrading packages, use:

package_upgrade: true
packages:
- nginx

 

Posted by Uli Köhler in Cloud, cloud-init

How to fix mongorestore ‘E11000 duplicate key error collection’

Problem:

You are trying to run mongorestore my-backup, but you see large numbers of warning messages like this one:

- E11000 duplicate key error collection: mydb.mycollection index: _id_ dup key: { : "MyKey" }

Solution:

By default, mongorestore does not overwrite or delete any existing documents. You need to tell it to drop each collection immediately before importing it from the backup:

mongorestore --drop my-backup

Any documents not in the backup will be permanently lost after running this!

Note that this will not drop collections that are not present in the backup.

Also see mongodump/mongorestore minimal examples

Posted by Uli Köhler in Databases

mongodump/mongorestore minimal examples

Create & restore a database

mongodump --db mydb --out mydb.mongobackup

This will backup the database mydb from the MongoDB running at localhost and store the backup in the newly created DigiKey.mongobackup directory as BSON.

mongorestore mydb.mongobackup

This will restore the backup to localhost (the database name, mydb, is stored in the backup directory).

It will not overwrite or update existing documents, nor delete documents that are currently present but not present in the backup.

Restore backup with drop

mongorestore --drop mydb.mongobackup

This will drop (i.e. delete) each collection before importing from the backup. This means that

  • Existing documents will effectively be overwritten
  • Documents that are currently present but not present in the backup will be deleted

However note that while importing the backup, some documents might be missing from the database until the backup has been fully restored.

Note that this will not drop collections that are not present in the backup.

Posted by Uli Köhler in Databases

What is a ‘headless’ program or application?

If you’re a backend programmer, you most likely have encountered the term of headless applications (like headless Java or headless Chromium) many times. But what does headless actually mean?

headless means it runs without a graphical user interface (GUI). In most cases, headless applications are command line applications or applications that are interfaced from a programming language.

When you open your browser on your Desktop or Laptop, you’ll see the browser window pop up. This window is a graphical user interface (GUI). Unless you are living under a rock, you have had your fair share of experience with GUIs and know how convenient they can be sometimes.

However, especially in the professional IT world, command line user interfaces are also common – mostly, because they can be used on servers and local PCs with a screen alike, and they are much easier to automate than GUIs.

If you have worked in application development for some time, you might know that GUI applications require a huge number of incredibly complex libraries to run. On Linux that includes the X11 server (X11 is the system that displays and arranges all the windows and other graphical features on your screen), some utility libraries for X11 and possibly some libraries to be able to create the high-level user interfaces in use in most applications today: While high-level GUIs consist of buttons and text inputs, low-level GUIs consist of pixel, colored areas and callback functions (from which the buttons and text inputs are built of).

As said before, headless applications run without a GUI. Under the right circumstances, this has two main advantages:

  • You don’t have to spend as much time developing a complex graphical user interface
  • You can easily run it on a server
Why can you run headless applications on a server but not normal GUI applications?

Firstly, as we said before, GUIs require a large number of complex libraries and software infrastructure to be present on a server. On many servers (like most Linux servers), this software is not installed by default – because installing it would eat up valuable resources like system memory (RAM), hard drive space and the system would be hard to maintain.

Contrary to common belief, running a GUI infrastructure on Linux (i.e. X server plus some utilities) does not require a screen or a dedicated graphics card. See our post on How to run X server using xserver-xorg-video-dummy driver on Ubuntu for an example on how to accomplish this.

Secondly, GUIs are hard to monitor, maintain and automate:

Imagine you have 25 servers running the same application on each of them. If that were a GUI application, you would have to look at 25 windows displaying the state of the application – not to mention the time spent to make all the windows display on your local screen reliably – or the development time spent to make the GUI

Using headless applications you can instead easily automate the task of monitoring the applications and therefore just display a summary on your local screen. Since you don’t have to spend time writing and maintaining GUIs, you can also spend your time more wisely.

Posted by Uli Köhler in Technologies