%%sitename%%

Aug

2019

How to intercept AJAX JSON response in Pyppeteer

This example shows you how to intercept and print the content of a JSON response requested via any AJAX request on a web page by using Pyppeteer:

import asyncio
import json
from pyppeteer import launch

async def intercept_network_response(response):
    # In this example, we care only about responses returning JSONs
    if "application/json" in response.headers.get("content-type", ""):
        # Print some info about the responses
        print("URL:", response.url)
        print("Method:", response.request.method)
        print("Response headers:", response.headers)
        print("Request Headers:", response.request.headers)
        print("Response status:", response.status)
        # Print the content of the response
        try:
            # await response.json() returns the response as Python object
            print("Content: ", await response.json())
        except json.decoder.JSONDecodeError:
            # NOTE: Use await response.text() if you want to get raw response text
            print("Failed to decode JSON from", await response.text())

async def main():
    browser = await launch()
    page = await browser.newPage()
    
    page.on('response', intercept_network_response)
            
    await page.goto('https://instagram.com')
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

Aug

2019

Pyppeteer minimal network response interception example

Using Javascript (puppeteer)? Check out Minimal puppeteer response interception example

This example shows you how to intercept network responses in pyppeteer.

Note: This intercepts the response, not the request! This means you can abort the request before it is actually sent to the server, but you can’t read the content of the response! See Pyppetteer minimal network request interception example for an example on how to intercept requests.

import asyncio
from pyppeteer import launch

async def intercept_network_response(response):
    # In this example, we only care about HTML responses!
    if "text/html" in response.headers.get("content-type", ""):
        # Print some info about the responses
        print("URL:", response.url)
        print("Method:", response.request.method)
        print("Response headers:", response.headers)
        print("Request Headers:", response.request.headers)
        print("Response status:", response.status)
        # Print the content of the response
        print("Content: ", await response.text())
        # NOTE: Use await response.json() if you want to get the JSON directly

async def main():
    browser = await launch()
    page = await browser.newPage()
    
    page.on('response', intercept_network_response)
            
    await page.goto('https://techoverflow.net')
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

Aug

2019

How to fix Python NameError: name ‘reponse’ is not defined

If you see this error message in Python:

NameError: name 'reponse' is not defined

you likely just mis-spelled response !

reponse  <-- your code, you're missing an 's'!
response <-- correct

Aug

2019

Pyppeteer minimal network request interception example

Using Javascript (puppeteer)? Check out Minimal puppeteer request interception example

This example shows you how to intercept network requests in pyppeteer:

Note: This intercepts the request, not the response! This means you can abort the request made, but you can’t read the content of the response! See Pyppetteer minimal network response interception example for an example on how to intercept responses.

import asyncio
from pyppeteer import launch

async def intercept_network_request(request):
    # Print some info about the request
    print("URL:", request.url)
    print("Method:", request.method)
    print("Headers:", request.headers)
    # NOTE: You can also await request.abort() to abort the requst1
    await request.continue_()

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.setRequestInterception(True)
    
    page.on('request', intercept_network_request)
            
    await page.goto('https://techoverflow.net')
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

Aug

2019

How to fix pyppeteer.errors.NetworkError: Request interception is not enabled.

Note: Also see Pyppetteer minimal network request interception example

Problem:

You are trying to intercept a request in Pyppeteer using

page.on('request', my_intercept_request)

but you’re getting an error message like this:

Traceback (most recent call last):
  File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/dist-packages/pyee/_compat.py", line 62, in _callback
    self.emit('error', exc)
  File "/usr/local/lib/python3.6/dist-packages/pyee/_base.py", line 106, in emit
    self._emit_handle_potential_error(event, args[0] if args else None)
  File "/usr/local/lib/python3.6/dist-packages/pyee/_base.py", line 83, in _emit_handle_potential_error
    raise error
  File "run.py", line 6, in intercept
    await request.continue_()
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/network_manager.py", line 481, in continue_
    raise NetworkError('Request interception is not enabled.')
pyppeteer.errors.NetworkError: Request interception is not enabled.

Solution:

Add

await page.setRequestInterception(True)

directly after your call to

page = await browser.newPage()

This will enable request interception and your code will run just fine.

Aug

2019

How to fix PyVISA “No module named ‘serial.tools'”

Problem:

You want to use an ASRL (serial) instrument in PyVISA, but when you run

python3 -m visa info

you get this output even though you have serial installed:

ASRL INSTR:
   Please install PySerial (>=3.0) to use this resource type.
   No module named 'serial.tools'

Solution:

You have installed serial but you need to install pyserial – they are not the same!

First you need to remove the system package python3-serial if installed. Example for Ubuntu/Debian:

sudo apt remove python3-serial

and also remove the pip serial package if installed

sudo pip3 uninstall serial

Then install pyserial:

sudo pip3 install pyserial

You can check if PySerial is installed properly using

python3 -m visa info

It should show you

ASRL INSTR: Available via PySerial (3.4)

once pyserial is installed correctly!

Note: The commands above are for Python 3.x. In case you are still using Python 2.x use pip2 or pip instead of pip3 and use python-serial instead of python3-serial as APT package name.

Aug

2019

How to fix PyVISA not finding any ASRL (serial port) instruments

Problem:

You are trying to connect to a USB instrument using PyVISA & pyvisa-py, but the PyVISA resource manager doesn’t find any instruments:

#!/usr/bin/env python3
import visa
rm = visa.ResourceManager()
print(rm.list_resources()) # Prints "()" => No instruments found!

Solution:

Install PySerial 3.0+:

First you need to remove the system package python3-serial if installed. Example for Ubuntu/Debian:

sudo apt remove python3-serial

and also remove the pip serial package if installed (we need to install pyserial, not serial!)

sudo pip3 uninstall serial

Then install pyserial:

sudo pip3 install pyserial

You can check if PySerial is installed properly using

python3 -m visa info

It should show you

ASRL INSTR: Available via PySerial (3.4)

if pyserial is installed correctly!

Note: The commands above are for Python 3.x. In case you are still using Python 2.x use pip2 or pip instead of pip3 and use python-serial instead of python3-serial as APT package name.

Aug

2019

How to fix PyVISA ‘Found a device whose serial number cannot be read. The partial VISA resource name is: USB0::[…]::[…]::???::0::INSTR’

Problem:

You are trying to list available resources using PyVISA e.g. using

import visa
rm = visa.ResourceManager()
print(rm.list_resources())

But when you try to run it, you see an output like

Found a device whose serial number cannot be read. The partial VISA resource name is: USB0::6833::3601::???::0::INSTR
()

Solution:

Even though PyVISA doesn’t tell you that exactly, this is just the bog-standard Linux USB permission problem. We already provided a generic solution in How to fix ALL USB permission issues on Linux once and for all.

Excerpt from this post (see there for details on why it works):

Run this in your favourite shell:

wget https://techoverflow.net/scripts/udev-install-usbusers.sh | sudo bash -s $USER

This will print:

SUBSYSTEM=="usb", MODE="0666", GROUP="usbusers"
USB device configuration has been installed. Please log out and log back in or reboot

then log out and log back in (or close your SSH session and log back in).

In case this doesn’t work, reboot!

After that, your PyVISA script should work as intended and should print e.g.

('USB0::6833::3601::DL3A204800938::0::INSTR')

Aug

2019

How to fix ALL USB permission issues on Linux once and for all

On Linux users often have the issue that normal users can’t access some USB devices while root can. Most pages on the internet try to address this issue individually for each device, but most users don’t need that granularity, they just want it to work.

This post provides a method

Installation

Run this in your favourite shell:

wget https://techoverflow.net/scripts/udev-install-usbusers.sh | sudo bash -s $USER

This will print:

SUBSYSTEM=="usb", MODE="0666", GROUP="usbusers"
USB device configuration has been installed. Please log out and log back in or reboot

then log out and log back in (or close your SSH session and log back in).

In case this doesn’t work, reboot!

How it works

It creates a group called usbusers
It adds your user ($USER) to the usbusers group. You might need to sudo usermod -a -G usbusers $USER for additional users that should have access to USB devices!
Then it creates an udev config file /etc/udev/rules.d/99-usbusers.rules with the following content:
```
SUBSYSTEM=="usb", MODE="0666", GROUP="usbusers"
```
It then tries to reload & trigger udev using udevadm. This usually means you don’t have to reboot

In effect, it sets the group to usbusers for every USB device, no matter what type and ensures the group has write access. This is why this solution is so generic – it’s not limited to a specific type of USB device.

Aug

2019

How to pad string with zeroes & right-justify in bash using sed

Pad any string using this sed command:

sed -e :a -e 's/^.\{1,3\}$/0&/;ta'

This pads every string to length 4, usage example:

$ echo x | sed -e :a -e 's/^.\{1,3\}$/0&/;ta'
000x

In case you want to pad to a different length, replace 3 in the script by (your desired length - 1).

You can also use a bash function like this:

# Zero pad to length 4, right-justified
function zero_pad4 { echo $1 | sed -e :a -e 's/^.\{1,3\}$/0&/;ta' ; }

Usage example:

$ zero_pad4 1x
001x

Original source, modified: The Unix School

Aug

2019

How to fix PyVISA not finding any USB instruments

Problem:

You are trying to connect to a USB instrument using PyVISA & pyvisa-py, but the PyVISA resource manager doesn’t find any instruments:

#!/usr/bin/env python3
import visa
rm = visa.ResourceManager()
print(rm.list_resources()) # Prints "()" => No instruments found!

Solution:

In order for pyvisa-py to be able to connect to USB instruments, you need to install the Python usb library!

On Debian or Ubuntu, install it using

sudo apt-get -y install python3-usb

or, if you are still using Python 2.x

sudo apt-get -y install python-usb

Now, re-run the script – you should see an output like

('USB0::6833::3601::DL3A204800938::0::INSTR',)

In case you still don’t see the output, run python3 -m visa info or python -m visa info (for Python 2.x).

It should show an output like this:

Machine Details:
   Platform ID:    Linux-4.19.0-5-686-i686-with-debian-10.0
   Processor:      

Python:
   Implementation: CPython
   Executable:     /usr/bin/python3
   Version:        3.7.3
   Compiler:       GCC 8.3.0
   Bits:           32bit
   Build:          Apr  3 2019 05:39:12 (#default)
   Unicode:        UCS4

PyVISA Version: 1.9.1

Backends:
   ni:
      Version: 1.9.1 (bundled with PyVISA)
      Binary library: Not found
   py:
      Version: 0.3.1
      ASRL INSTR: Available via PySerial (3.4)
      USB INSTR: Available via PyUSB (1.0.2). Backend: libusb1
      USB RAW: Available via PyUSB (1.0.2). Backend: libusb1
      TCPIP INSTR: Available 
      TCPIP SOCKET: Available 
      GPIB INSTR:
         Please install linux-gpib to use this resource type.
         No module named 'gpib'

Check Backends -> py -> USB INSTR: If it’s not Available via PyUSB, check the information message for hints what might be the issue. For example, if it says

USB INSTR:
   Please install PyUSB to use this resource type.
   No module named 'usb'

that means that the Python USB library has not been installed properly.

If USB is Available via PyUSB but PyVISA still doesn’t find the instrument, check if it is connected properly using

lsusb

which should show a line related to your instrument’s manufacturer, e.g.

Bus 001 Device 002: ID 1ab1:0e11 Rigol Technologies

Also unplug and re-plug your instrument so Linux tries to reconnect to the USB device and check the end of the output of sudo dmesg which could list e.g.

[19427.230120] usb 1-2: new high-speed USB device number 2 using ehci-pci
[19427.425464] usb 1-2: config 1 interface 0 altsetting 0 bulk endpoint 0x82 has invalid maxpacket 64
[19427.425469] usb 1-2: config 1 interface 0 altsetting 0 bulk endpoint 0x3 has invalid maxpacket 64
[19427.425947] usb 1-2: New USB device found, idVendor=1ab1, idProduct=0e11, bcdDevice= 0.02
[19427.425950] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[19427.425953] usb 1-2: Product: DL3000 Serials
[19427.425955] usb 1-2: Manufacturer: Rigol Technologies. 
[19427.425957] usb 1-2: SerialNumber: DL3A204800938
[19429.525745] usbcore: registered new interface driver usbtmc

usbtmc in the last line means that the USB device has been recognized as USB Test & Measurement class device, and hence you should be able to connect to it using PyVISA as USB INSTR.

Aug

2019

How to fix PyVISA ‘ValueError: Could not locate a VISA implementation. Install either the NI binary or pyvisa-py.’

Problem:

You are trying to use PyVISA to connect to an instrument, but you see an error message like

Traceback (most recent call last):
  File "TestPyVISA.py", line 3, in <module>
    rm = visa.ResourceManager()
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/highlevel.py", line 1526, in __new__
    visa_library = open_visa_library(visa_library)
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/highlevel.py", line 1493, in open_visa_library
    wrapper = _get_default_wrapper()
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/highlevel.py", line 1470, in _get_default_wrapper
    raise ValueError('Could not locate a VISA implementation. Install either the NI binary or pyvisa-py.')
ValueError: Could not locate a VISA implementation. Install either the NI binary or pyvisa-py.

Solution:

Install the pyvisa-py Python package:

sudo pip3 install pyvisa-py

or, if you are still using Python 2.x:

sudo pip2 install pyvisa-py

Aug

2019

Capacitor energy from capacitance and voltage online calculator & Python code

TechOverflow calculators:
You can enter values with SI suffixes like 12.2m (equivalent to 0.012) or 14k (14000) or 32u (0.000032).
The results are calculated while you type and shown directly below the calculator, so there is no need to press return or click on a Calculate button. Just make sure that all inputs are green by entering valid values.

C ⚠

V ⚠

Formula

E = \frac{1}{2}\cdot{}C\cdot{}U_p^2

Python code

You can use the UliEngineering library like this:

from UliEngineering.Electronics.Capacitors import capacitor_energy
from UliEngineering.EngineerIO import auto_format, auto_print

# These are equivalent:
energy = capacitor_energy("100 uF", "24 V") # energy = 0.0288 (J)
energy = capacitor_energy(100e-6, 24.0) # energy = 0.0288 (J)

# ... or get out a human-readable value:
energy_str = auto_format(capacitor_energy, "100 uF", "24 V") # "28.8 mJ"
# ... or print directly
auto_print(capacitor_energy, "100 uF", "24 V") # prints "28.8 mJ"

In case you can’t use UliEngineering, use this Python function:

def capacitor_energy(capacitance, voltage):
    return 0.5*capacitance*voltage*voltage

# Usage example:
print(capacitor_energy(100e-6, 24.0)) # prints 0.0288 (J)

Aug

2019

X1/X2/Y1/Y2/Y4 impulse withstand rating voltage calculator (IEC 60384-14)

IEC 60384-14 specifies that X1/X2-rated capacitors shall be tested to withstand an impulse voltage of 4 kV (X1), 2.5 kV (X2, Y4), 8 kV (Y1) or 5 kV (Y2).

However these values only apply for a capacitance $\leq 1 μF$ (except for Y1/Y4 capacitors). Use this calculator for X1/X2/Y2 capacitances $> 1 μF$ !

F ⚠

Formula:

Up = \frac{Up_{\leq 1 μF}}{\Large\sqrt{\frac{C}{1\,000\,000\frac{μF}{F}}}}

where:

$Up$ is the impulse withstand voltage rating
$C$ is the capacitance in Farads
Up_{\leq 1 μF} is the voltage rating for that capacitor class with a capacitance of \leq 1 μF:
- For X1-class: 4 kV
- For X2-class: 2.5 kV
- Y1-class impulse withstand voltage is always 8 kV no matter what capacitance
- For Y2-class: 5 kV
- Y4-class impulse withstand voltage is always 2.5 kV no matter what capacitance

Why is the impulse withstand voltage lower for larger capacitors?

The rationale behind the derating of the impulse withstand voltage is that larger capacitances will have sufficient capacitance so that a given overvoltage doesn’t cause a large voltage spike in the capacitor.

The formula (see above) is chosen so that the energy in the capacitor:

E = \frac{1}{2}\cdot{}C\cdot{}U_p^2

is kept constant (i.e. at the same value as for a equivalent capacitor of 1 μF).

Aug

2019

How to fix pyaudio ‘fatal error: portaudio.h: No such file or directory’

Problem:

You are trying to install pyaudio using sudo pip install pyaudio or a similar command but you see an error message like

    src/_portaudiomodule.c:29:23: fatal error: portaudio.h: No such file or directory
     #include "portaudio.h"
                           ^
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-jgxnwixs/pyaudio/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-c3blzlv5-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-jgxnwixs/pyaudio/

Solution:

You need to install the portaudio library. On Debian/Ubuntu you can do that using

sudo apt install portaudio19-dev

on other systems either search for portaudio in your package manager or download the library from http://www.portaudio.com/

Aug

2019

What is ‘…’ in HTML?

… is the HTML entity code for the ellipsis dots:

…

If you see it on a webpage outside the HTML source code, it usually means that the author or content management system has encoded the HTML entity incorrectly. You should only ever see … on the page except in literal HTML sourcecode.

I recommend using the unicode character HORIZONTAL ELLIPSIS (see the code block above) instead of the HTML entity …. Ensure that you have

<meta charset="utf-8" />

in your HTML <head> for compatibility.

Aug

2019

What is performance criterion A/B/C/D in EMI tests?

When doing EMI immunity tests, you will often encounter specifications that your device should e.g. “meet Performance Criterion B during the ESD test”.

The performance criteria, as defined in the IEC 61000 series of standards, e.g. IEC 61000-4-5:2014 (surge test standard) are:

Performance Criterion A: Your device continues to function normally during the test (i.e. no interruption)
Performance Criterion B: Your device shows a malfunction but recovers automatically (without user interaction) and then continues to operate normally.
Performance Criterion C: Your devices shows a malfunction, but continues to operate normally only after user interaction (e.g. the user resets the device)
Performance Criterion D: Your device has a permanent malfunction (e.g. some part of its hardware has been destroyed, or data being lost) which cannot be recovered from even after user interaction.

Additionally note that your device must not become dangerous under any circumstances, even if destroyed as per Performance Criterion D.

What defines a malfunction is up to the manufacturer of the device, but usually it’s related to whatever function it is advertised to perform. The manufacturer should define

Example: 230V LED bulb

Performance Criterion A: LED Bulb continues to operate normally after the test
Performance Criterion B: Bulb flickers during the test, but operates normally after the test
Performance Criterion B: Bulb is less bright during the test but then operates normally.
Performance Criterion B: Bulb blinks 10 times slowly but then operates normally
Performance Criterion B: Bulb stays off for 20 seconds after the test but then goes back to normal.
Performance Criterion C: Bulb goes out, but works normally after you switch off and on the power
Performance Criterion C: Bulb continues to blink, but works normally after you switch off and on the power
Performance Criterion D: Bulb goes out and does not turn on after you switch off and on the power
Performance Criterion D: Bulb blinks after the test and continues to blink even if you switch off and on the power.

As a manufacturer you could define e.g.

that the light intensity should be judged by eye (i.e. it should not be significantly dimmer after the test than a new bulb)
that a luxmeter needs to be used to judge the light intensity after tests
that the bulb may stay off for at most 15 seconds

and so on. In all but the most demanding applications, manufacturers define the criteria conservatively in order to increase the likelihood of passing the test. Since EMI laboratory tests are expensive, it’s in your best interest as a manufacturer not to over-define the criteria.

Aug

2019

How to fix LaTeX missing footnotes in tabular environment

Problem:

You have LaTeX code with a \footnote inside a \tabular like this:

\documentclass{scrartcl}

\begin{document}
\begin{tabular}{p{2cm}|p{2cm}|}
\textbf{Column A\footnote{This is a footnote!}} & \textbf{Column B}\\
1 & 2\\
3 & 4\\
\end{tabular}
\end{document}

but LaTeX/PDFLaTeX is not showing any of the footnotes you declared inside the \tabular environment.

Solution:

Use this code just after your \documentclass declaration:

\usepackage{footnote}
\makesavenoteenv{tabular}

Then recompile your LaTeX code. The footnotes inside your tabular should now appear as expected.

Note that if you are using other environments than \begin{tabular} you might need to add more \makesavenoteenv declarations for the correct environments. The tabularx environment from the tabularx package works with footnotes out-of-the box without any additional packages!

Jul

2019

How to fix LaTeX \glqq or \grqq error ‘Undefined control sequence’

Problem:

You have LaTeX code containing \glqq and/or \grqq like

\documentclass{scrartcl}

\begin{document}
\glqq Test\grqq
\end{document}

but when you try to compile it, you see an error message like this:

! Undefined control sequence.
l.5 \glqq
          Test\grqq
?

Solution:

In order to use \glqq or \grqq you need to include

\usepackage[ngerman]{babel}

after your \documentclass declaration.

In case you want to use another language, use the correct language specifier instead of ngerman.

Jul

2019

How to create a self-deleting temporary directory in Python

Starting from Python 3.2 your can use tmpfile.TemporaryDirectory like this:

from tempfile import TemporaryDirectory

with TemporaryDirectory(prefix="myapp-") as tmpdir:
    print(tmpdir) # e.g. "/tmp/myapp-fevhzh93"

# Once you are outside the "with" block, the directory will be deleted!

In case you are stuck with using older Python versions, check out How to create temporary directory in Python.

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Installation

How it works

Problem:

Solution:

Problem:

Solution:

Formula

Python code

Formula:

Why is the impulse withstand voltage lower for larger capacitors?

Problem:

Solution:

Example: 230V LED bulb

Problem:

Solution:

Problem:

Solution:

Search

Categories