Javascript

Aug

2019

Minimal puppeteer request interception example

Using Python (pyppeteer)? Check out Pyppeteer minimal network request interception example

This example shows you how to intercept network requests in puppeteer:

Note: This intercepts the request, not the response! This means you can abort the request made, but you can’t read the content of the response! See Minimal puppeteer response interception example for an example on how to intercept responses.

const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  // Enable request interception
  await page.setRequestInterception(true);
  page.on('request', async (request) => {
      console.info("URL", request.url());
      console.info("Method", request.method())
      console.info("Headers", request.headers())
      return request.continue(); // Allow request to continue
      // return request.abort(); // use this instead to abort the request!
  })
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Make a screenshot
  await page.screenshot({path: 'screenshot.png'});
  await browser.close();
})();

Aug

2019

Javascript equivalent to Python’s re.findall()

The equivalent to this Python code which is using re.findall()

import re

hashtag_regex = r"(\B#\w\w+)"
hits = re.findall(hashtag_regex, "This is a string #with #hashtags")
print(hits) # prints ['#with', '#hashtags']

in Javascript is

const string = "This is a string #with #hashtags";
const re = /(\B#\w\w+)/g;
const hits = [];
// Iterate hits
let match = null;
do {
    match = re.exec(string);
    if(match) {
        hits.push(match[0]);
    }
} while (match);

console.log(hits); // Prints [ '#with', '#hashtags' ]

You need to assign your regular expression to a variable like re! If you use

match = /(\B#\w\w+)/g.exec(string); // WRONG ! Don't do this !

you will create an infinite loop which always generates the first hit in the string, if any!

Aug

2019

How to fix Puppeteer ‘Error: Protocol error (Emulation.setDeviceMetricsOverride): Invalid parameters width: integer value expected; height: integer value expected’

Problem:

You are trying to set the viewport size in Puppeteer using

const browser = await puppeteer.launch({
    defaultViewport: '1920x1080'
});

but you see an error message like

(node:15692) UnhandledPromiseRejectionWarning: Error: Protocol error (Emulation.setDeviceMetricsOverride): Invalid parameters width: integer value expected; height: integer value expected
    at /home/uli/dev/myproject/node_modules/puppeteer/lib/Connection.js:183:56
    at new Promise (<anonymous>)
    at CDPSession.send (/home/uli/dev/myproject/node_modules/puppeteer/lib/Connection.js:182:12)
    at EmulationManager.emulateViewport (/home/uli/dev/myproject/node_modules/puppeteer/lib/EmulationManager.js:41:20)
    at Page.setViewport (/home/uli/dev/myproject/node_modules/puppeteer/lib/Page.js:808:54)
    at Page.<anonymous> (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:112:23)
    at Function.create (/home/uli/dev/myproject/node_modules/puppeteer/lib/Page.js:49:18)
    at processTicksAndRejections (internal/process/task_queues.js:85:5)
    at async Browser._createPageInContext (/home/uli/dev/myproject/node_modules/puppeteer/lib/Browser.js:177:18)
    at async /home/uli/dev/myproject/index.js:8:16
  -- ASYNC --
    at Target.<anonymous> (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:111:15)
    at Browser._createPageInContext (/home/uli/dev/myproject/node_modules/puppeteer/lib/Browser.js:177:31)
    at processTicksAndRejections (internal/process/task_queues.js:85:5)
    at async /home/uli/dev/myproject/index.js:8:16
  -- ASYNC --
    at Browser.<anonymous> (/home/uli/dev/myproject/node_modules/puppeteer/lib/helper.js:111:15)
    at /home/uli/dev/myproject/index.js:8:30
    at processTicksAndRejections (internal/process/task_queues.js:85:5)
(node:15692) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:15692) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Solution:

You need to use

{defaultViewport: {width: 1920, height: 1080}}

instead of

{defaultViewport: '1920x1080'}

Full example:

// Minimal puppeteer example
const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch({
      defaultViewport: {width: 1920, height: 1080}
  });
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Make a screenshot
  await page.screenshot({path: 'screenshot.png'});
  await browser.close();
})();

Aug

2019

How to set screenshot size in Puppeteer?

Problem:

You are trying to make a screenshot using Puppeteer like this:

await page.screenshot({path: 'screenshot.png'});

but the screenshot is always 800×600 pixels. How can you increase the size of the screenshot?

Solution:

You can’t set the size for the screenshot, you need to set the size for the viewport in puppeteer.launch:

const browser = await puppeteer.launch({
    defaultViewport: {width: 1920, height: 1080}
});

Full example to produce a 1920x1080 screenshot:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
      defaultViewport: {width: 1920, height: 1080}
  });
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  // Wait until page has loaded completely
  // Make a screenshot
  await page.screenshot({path: 'screenshot.png'});

  await browser.close();
})();

Aug

2019

Puppeteer minimal example

You can use this example as a starting point for your puppeteer application.

// Minimal puppeteer example
const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://techoverflow.net', {waitUntil: 'domcontentloaded'});
  //
  // YOUR CODE GOES HERE!
  //
  await browser.close();
})();

The screenshot is 800x600px by default and could look like this:

Aug

2019

Pure Javascript equivalent to jQuery $(window).load()

The pure Javascript equivalent to

$(window).load(function() {
    // Your code goes here!

})

window.addEventListener("load", function() { 
    // Your code goes here!
});

Note that it’s not supported by IE8, but that should not be an issue in 2019.

Jul

2019

Pure Javascript equivalent to jQuery $(document).ready()

The pure Javascript equivalent to

$(document).ready(function() {
    // Your code goes here!

})

document.addEventListener("DOMContentLoaded", function() { 
    // Your code goes here!
});

Note that it’s not supported by IE8, but that should not be an issue in 2019.

May

2019

How to install NodeJS 14.x LTS on Ubuntu in 1 minute

Newer version 16 available: How to install NodeJS 16.x LTS on Ubuntu in 1 minute

Run these shell commands on your Ubuntu computer to install NodeJS 14.x:

curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
sudo apt-get install -y nodejs

Instead of setup_14.x you can also choose other versions like setup_12.x. However, using this method, you can’t install multiple versions of NodeJS in parallel.

Source: Official nodesource documentation

May

2019

How to fix ‘ng upgrade’ error ‘The specified command (“upgrade”) is invalid’

Problem:

You want to run ng upgrade to update your angular project libraries, but you see this error message:

The specified command ("upgrade") is invalid. For a list of available options,
run "ng help".

Did you mean "update"?

Solution:

You need to run

ng update

instead of ng upgrade (which is not a valid command).

This is what Did you mean "update"? in the error message is intended to point you towards.

Apr

2019

How to fix npm ERR! missing script: start

Problem:

You want to run a NodeJS app using

npm start

but you only see this error message:

npm ERR! missing script: start

Solution:

You need to tell npm what to do when you run npm start explicitly by editing package.json.

First, identify the main file of your application. Most often it is called index.js, server.js or app.js. When you open package.json in an editor, you can also often find a line like

"main": "index.js",

In this example, index.js is the main file of the application.

Now we can edit package.json to add a start script.

In package.json, find the "scripts" section. If you have a default package.json, it will look like this:

"scripts": {
  "test": "echo \"Error: no test specified\" && exit 1"
},

Now add a comma to the end of the "test" line and add this line after it:

"start": "node index.js"

and replace index.js by the main file of your application (e.g. app.js, server.js, index.js etc).

The "scripts" section should now look like this:

"scripts": {
  "test": "echo \"Error: no test specified\" && exit 1",
  "start": "node index.js"
},

Now save and close package.json and run

npm start

to start your application.

Apr

2019

Koa minimal example

Also see Minimal Koa.JS example with Router & Body parser

const Koa = require('koa');
const app = new Koa();

app.use(async ctx => {
  ctx.body = {status: 'success'};
});

app.listen(8080);

To install (see this previous post for a guide on installing NodeJS & NPM):

npm install --save koa
node index.js

Then go to http://localhost:8080/ to see the test page.

Mar

2019

MongoDB: How to run db.adminCommand() in NodeJS

Problem:

You want to run a db.adminCommand() in NodeJS using the node-mongodb-native client, e.g. you want to run the NodeJS equivalent of

db.adminCommand({setParameter: 1, internalQueryExecMaxBlockingSortBytes: 100151432});

Solution:

Use conn.executeDbAdminCommand() where db is a MongoDB database object.

db.executeDbAdminCommand({setParameter: 1, internalQueryExecMaxBlockingSortBytes: 100151432});

Full example:

// To install, use npm i --save mongodb
const MongoClient = require('mongodb').MongoClient;

async function configureMongoDB() {
    // Connect to MongoDB
    const conn = await MongoClient.connect('mongodb://localhost:27017/', { useNewUrlParser: true });
    const db = await conn.db('mydb');
    // Configure MongoDB settings
    await db.executeDbAdminCommand({
        setParameter: 1,
        internalQueryExecMaxBlockingSortBytes: 100151432
    });
    // Cleanup
    return conn.close();
}

// Run configureMongoDB()
configureMongoDB().then(() => {}).catch(console.error)

Mar

2019

How to get first character of String in Javascript

Use s.charAt(0)

Example:

const s = "banana";
const firstChar = s.charAt(0); // == 'b'
// This will print 'b'
console.log(firstChar);

Mar

2019

How to read NodeJS child-process.exec stdout/stderr using async/await Promises

You want to run a command like file my.pdf using NodeJS child-process.exec and get its stdout after it’s finished.

Solution:

TL;DR: (await exec('file my.pdf')).stdout

We’re using child-process-promise here in order to simplify our implementation. Install it using npm i --save child-process-promise !

const { exec } = require('child-process-promise');

async function run () {
    const ret = await exec(`file my.pdf`);
    return ret.stdout;
}

run().then(console.log).catch(console.error);

You can also use .stderr instead of .stdout to get the stderr output as a string

Mar

2019

How to fix NodeJS MongoDB ‘Cannot read property ‘high_’ of null’

When encountering an error message like

TypeError: Cannot read property 'high_' of null
    at Long.equals (/home/uli/dev/NMUN/node_modules/bson/lib/bson/long.js:236:31)
    at nextFunction (/home/uli/dev/NMUN/node_modules/mongodb-core/lib/cursor.js:473:16)
    at Cursor.next (/home/uli/dev/NMUN/node_modules/mongodb-core/lib/cursor.js:763:3)
    at Cursor._next (/home/uli/dev/NMUN/node_modules/mongodb/lib/cursor.js:211:36)
    at nextObject (/home/uli/dev/NMUN/node_modules/mongodb/lib/operations/cursor_ops.js:192:10)
    at hasNext (/home/uli/dev/NMUN/node_modules/mongodb/lib/operations/cursor_ops.js:135:3)
    (...)

you likely have code like this:

const cursor = db.getCollection('mycollection').find({})
while (cursor.hasNext()) {
    const doc = cursor.next();
    // ... handle doc ...
}

The solution is quite simple: Since find(), cursor.hasNext() and cursor.next() all return Promises, you can’t use their results directly.

This example shows you how to do it properly using async/await:

const cursor = await db.getCollection('mycollection').find({})
while (await cursor.hasNext()) {
    const doc = await cursor.next();
    // ... handle doc ...
}

In order to do this remember that the function containing this code will need to be an async function. See the Mozilla documentation or google for Javascript async tutorial in order to learn about the details!

Mar

2019

Fixing Promise ‘TypeError: myfunc(…).then(…).reject is not a function’

When encountering this error message in Javascript related to Promises:

TypeError: myfunc(...).then(...).reject is not a function

The solution is quite simple: When you want to catch the error from the promise, you need to use catch instead of reject like this:

myfunc(...).then(...).catch(...)

Feb

2019

How to install NodeJS 10.x on Ubuntu in 1 minute

Note: Also see How to install NodeJS 12.x on Ubuntu in 1 minute

Run these shell commands on your Ubuntu computer to install NodeJS 10.x:

curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
sudo apt-get install -y nodejs

Instead of setup_10.x you can also choose other versions like setup_8.x. However, using this method, you can’t install multiple versions of NodeJS in parallel.

Source: Official nodesource documentation

Jan

2019

Puppeteer: Get text content / inner HTML of an element

Problem:

You want to use puppeteer to automate testing a webpage. You need to get either the text or the inner HTML of some element, e.g. of

<div id="mydiv">
</div>

on the page.

Solution:

// Get inner text
const innerText = await page.evaluate(() => document.querySelector('#mydiv').innerText);

// Get inner HTML
const innerHTML = await page.evaluate(() => document.querySelector('#mydiv').innerHTML);

Note that .innerText includes the text of sub-elements. You can use the complete DOM API inside page.evaluate(...). You can use any CSS selector as an argument for document.querySelector(...).

Dec

2018

How to fix WebPack error describe: optionsSchema.definitions.output.properties.path.description

Problem:

You are trying to build your webpack project, but you see an error message like this:

/home/uli/project/node_modules/webpack-cli/bin/config-yargs.js:89
                                describe: optionsSchema.definitions.output.properties.path.description,
                                                                           ^

TypeError: Cannot read property 'properties' of undefined
    at module.exports (/home/uli/project/node_modules/webpack-cli/bin/config-yargs.js:89:48)
    at /home/uli/project/node_modules/webpack-cli/bin/webpack.js:60:27
    at Object.<anonymous> (/home/uli/project/node_modules/webpack-cli/bin/webpack.js:515:3)
    at Module._compile (internal/modules/cjs/loader.js:723:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:734:10)
    at Module.load (internal/modules/cjs/loader.js:620:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:560:12)
    at Function.Module._load (internal/modules/cjs/loader.js:552:3)
    at Module.require (internal/modules/cjs/loader.js:659:17)
    at require (internal/modules/cjs/helpers.js:22:18)

Solution:

This is a known bug in webpack 4.20.0 – you can circumvent this issue by using webpack 4.19.0.

Look for a line like

"webpack": "^4.7.0",

in your package.json. The caret (^) allows npm to use any 4.x.x version – including the broken 4.20.0.

Replace the aforementioned line by

"webpack": "4.19.0",

to use only webpack 4.19.0.

After that, run npm install and retry building your application.

Dec

2018

Fixing MomentJS interpreting dates as local time

Problem:

You live in a non-UTC timezone. Dates parsed using MomentJS like

moment("2017-01-01")

are interpreted as local time as opposed to UTC (as would be appropiate based on the Z in the ISO8601 timestamp) and therefore trying to format them yields an offset timestamp:

moment("2017-01-01").toDate().toISOString()
// "2016-12-31T23:00:00.000Z" <-- Offset by 1 hour (MEZ - UTC)

This causes you trouble as often the original date is not preserved: In the example above, the correct date would be 2017-01-01 but it is 2016-12-31 instead.

Solution 1: Force moment to parse the date as UTC

moment.utc("2017-01-01").toDate().toISOString()
// "2017-01-01T00:00:00.000Z" <-- Correct

Solution 2: Manually subtract the timezone offset

let m = moment("2017-01-01");
// Subtract the difference between timezone and UTC
m = m.subtract(m.toDate().getTimezoneOffset(), 'minutes');
// m.toDate().toISOString() === "2017-01-01T01:00:00.000Z" - CORRECT

Use this if Solution 1 does not work or you can’t modify the parsing code: This solution works even if the date has already been parsed.

DO NOT use

new Date().getTimezoneOffset() // NEVER USE THIS! MIGHT USE THE WRONG OFFSET!

because it uses the timezone offset of the current date instead of the date parsed by MomentJS. Obviously this is wrong in countries with summertime, since dates may have different offsets depending on the date. Moreover, in rare cases the client’s computer timezone may have changed due to travel etc in the meantime and might therefore might not represent the correct offset to use.

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Full example:

Example:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution 1: Force moment to parse the date as UTC

Solution 2: Manually subtract the timezone offset

Search

Categories