Python

How to intercept AJAX JSON response in Pyppeteer

This example shows you how to intercept and print the content of a JSON response requested via any AJAX request on a web page by using Pyppeteer:

import asyncio
import json
from pyppeteer import launch

async def intercept_network_response(response):
    # In this example, we care only about responses returning JSONs
    if "application/json" in response.headers.get("content-type", ""):
        # Print some info about the responses
        print("URL:", response.url)
        print("Method:", response.request.method)
        print("Response headers:", response.headers)
        print("Request Headers:", response.request.headers)
        print("Response status:", response.status)
        # Print the content of the response
        try:
            # await response.json() returns the response as Python object
            print("Content: ", await response.json())
        except json.decoder.JSONDecodeError:
            # NOTE: Use await response.text() if you want to get raw response text
            print("Failed to decode JSON from", await response.text())

async def main():
    browser = await launch()
    page = await browser.newPage()
    
    page.on('response', intercept_network_response)
            
    await page.goto('https://instagram.com')
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())
Posted by Uli Köhler in Puppeteer, Python

Pyppetteer minimal network response interception example

Using Javascript (puppeteer)? Check out Minimal puppeteer response interception example

This example shows you how to intercept network responses in pyppeteer.

Note: This intercepts the response, not the request! This means you can abort the request before it is actually sent to the server, but you can’t read the content of the response! See Pyppetteer minimal network request interception example for an example on how to intercept requests.

import asyncio
from pyppeteer import launch

async def intercept_network_response(response):
    # In this example, we only care about HTML responses!
    if "text/html" in response.headers.get("content-type", ""):
        # Print some info about the responses
        print("URL:", response.url)
        print("Method:", response.request.method)
        print("Response headers:", response.headers)
        print("Request Headers:", response.request.headers)
        print("Response status:", response.status)
        # Print the content of the response
        print("Content: ", await response.text())
        # NOTE: Use await response.json() if you want to get the JSON directly

async def main():
    browser = await launch()
    page = await browser.newPage()
    
    page.on('response', intercept_network_response)
            
    await page.goto('https://techoverflow.net')
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

 

Posted by Uli Köhler in Puppeteer, Python

Pyppetteer minimal network request interception example

Using Javascript (puppeteer)? Check out Minimal puppeteer request interception example

This example shows you how to intercept network requests in pyppeteer:

Note: This intercepts the request, not the response! This means you can abort the request made, but you can’t read the content of the response! See Pyppetteer minimal network response interception example for an example on how to intercept responses.

import asyncio
from pyppeteer import launch

async def intercept_network_request(request):
    # Print some info about the request
    print("URL:", request.url)
    print("Method:", request.method)
    print("Headers:", request.headers)
    # NOTE: You can also await request.abort() to abort the requst1
    await request.continue_()

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.setRequestInterception(True)
    
    page.on('request', intercept_network_request)
            
    await page.goto('https://techoverflow.net')
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

 

Posted by Uli Köhler in Puppeteer, Python

How to fix pyppeteer.errors.NetworkError: Request interception is not enabled.

Note: Also see Pyppetteer minimal network request interception example

Problem:

You are trying to intercept a request in Pyppeteer using

page.on('request', my_intercept_request)

but you’re getting an error message like this:

Traceback (most recent call last):
  File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/dist-packages/pyee/_compat.py", line 62, in _callback
    self.emit('error', exc)
  File "/usr/local/lib/python3.6/dist-packages/pyee/_base.py", line 106, in emit
    self._emit_handle_potential_error(event, args[0] if args else None)
  File "/usr/local/lib/python3.6/dist-packages/pyee/_base.py", line 83, in _emit_handle_potential_error
    raise error
  File "run.py", line 6, in intercept
    await request.continue_()
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/network_manager.py", line 481, in continue_
    raise NetworkError('Request interception is not enabled.')
pyppeteer.errors.NetworkError: Request interception is not enabled.

Solution:

Add

await page.setRequestInterception(True)

directly after your call to

page = await browser.newPage()

This will enable request interception and your code will run just fine.

Posted by Uli Köhler in Puppeteer, Python

How to fix PyVISA “No module named ‘serial.tools'”

Problem:

You want to use an ASRL (serial) instrument in PyVISA, but when you run

python3 -m visa info

you get this output even though you have serial installed:

ASRL INSTR:
   Please install PySerial (>=3.0) to use this resource type.
   No module named 'serial.tools'

Solution:

You have installed serial but you need to install pyserial – they are not the same!

First you need to remove the system package python3-serial if installed. Example for Ubuntu/Debian:

sudo apt remove python3-serial

and also remove the pip serial package if installed

sudo pip3 uninstall serial

Then install pyserial:

sudo pip3 install pyserial

You can check if PySerial is installed properly using

python3 -m visa info

It should show you

ASRL INSTR: Available via PySerial (3.4)

once pyserial is installed correctly!

Note: The commands above are for Python 3.x. In case you are still using Python 2.x use pip2 or pip instead of pip3 and use python-serial instead of python3-serial as APT package name.

Posted by Uli Köhler in Electronics, Python

How to fix PyVISA not finding any ASRL (serial port) instruments

Problem:

You are trying to connect to a USB instrument using PyVISA & pyvisa-py, but the PyVISA resource manager doesn’t find any instruments:

#!/usr/bin/env python3
import visa
rm = visa.ResourceManager()
print(rm.list_resources()) # Prints "()" => No instruments found!

Solution:

Install PySerial 3.0+:

First you need to remove the system package python3-serial if installed. Example for Ubuntu/Debian:

sudo apt remove python3-serial

and also remove the pip serial package if installed (we need to install pyserial, not serial!)

sudo pip3 uninstall serial

Then install pyserial:

sudo pip3 install pyserial

You can check if PySerial is installed properly using

python3 -m visa info

It should show you

ASRL INSTR: Available via PySerial (3.4)

if pyserial is installed correctly!

Note: The commands above are for Python 3.x. In case you are still using Python 2.x use pip2 or pip instead of pip3 and use python-serial instead of python3-serial as APT package name.

Posted by Uli Köhler in Electronics, Python

How to fix PyVISA ‘Found a device whose serial number cannot be read. The partial VISA resource name is: USB0::[…]::[…]::???::0::INSTR’

Problem:

You are trying to list available resources using PyVISA e.g. using

import visa
rm = visa.ResourceManager()
print(rm.list_resources())

But when you try to run it, you see an output like

Found a device whose serial number cannot be read. The partial VISA resource name is: USB0::6833::3601::???::0::INSTR
()

Solution:

Even though PyVISA doesn’t tell you that exactly, this is just the bog-standard Linux USB permission problem. We already provided a generic solution in How to fix ALL USB permission issues on Linux once and for all.

Excerpt from this post (see there for details on why it works):

Run this in your favourite shell:

wget https://techoverflow.net/scripts/udev-install-usbusers.sh | sudo bash -s $USER

This will print:

SUBSYSTEM=="usb", MODE="0666", GROUP="usbusers"
USB device configuration has been installed. Please log out and log back in or reboot

then log out and log back in (or close your SSH session and log back in).

In case this doesn’t work, reboot!

After that, your PyVISA script should work as intended and should print e.g.

('USB0::6833::3601::DL3A204800938::0::INSTR')

 

Posted by Uli Köhler in Electronics, Python

How to fix PyVISA not finding any USB instruments

Problem:

You are trying to connect to a USB instrument using PyVISA & pyvisa-py, but the PyVISA resource manager doesn’t find any instruments:

#!/usr/bin/env python3
import visa
rm = visa.ResourceManager()
print(rm.list_resources()) # Prints "()" => No instruments found!

Solution:

In order for pyvisa-py to be able to connect to USB instruments, you need to install the Python usb library!

On Debian or Ubuntu, install it using

sudo apt-get -y install python3-usb

or, if you are still using Python 2.x

sudo apt-get -y install python-usb

Now, re-run the script – you should see an output like

('USB0::6833::3601::DL3A204800938::0::INSTR',)

In case you still don’t see the output, run python3 -m visa info or python -m visa info (for Python 2.x).

It should show an output like this:

Machine Details:
   Platform ID:    Linux-4.19.0-5-686-i686-with-debian-10.0
   Processor:      

Python:
   Implementation: CPython
   Executable:     /usr/bin/python3
   Version:        3.7.3
   Compiler:       GCC 8.3.0
   Bits:           32bit
   Build:          Apr  3 2019 05:39:12 (#default)
   Unicode:        UCS4

PyVISA Version: 1.9.1

Backends:
   ni:
      Version: 1.9.1 (bundled with PyVISA)
      Binary library: Not found
   py:
      Version: 0.3.1
      ASRL INSTR: Available via PySerial (3.4)
      USB INSTR: Available via PyUSB (1.0.2). Backend: libusb1
      USB RAW: Available via PyUSB (1.0.2). Backend: libusb1
      TCPIP INSTR: Available 
      TCPIP SOCKET: Available 
      GPIB INSTR:
         Please install linux-gpib to use this resource type.
         No module named 'gpib'

Check Backends -> py -> USB INSTR: If it’s not Available via PyUSB, check the information message for hints what might be the issue. For example, if it says

USB INSTR:
   Please install PyUSB to use this resource type.
   No module named 'usb'

that means that the Python USB library has not been installed properly.

If USB is Available via PyUSB but PyVISA still doesn’t find the instrument, check if it is connected properly using

lsusb

which should show a line related to your instrument’s manufacturer, e.g.

Bus 001 Device 002: ID 1ab1:0e11 Rigol Technologies

Also unplug and re-plug your instrument so Linux tries to reconnect to the USB device and check the end of the output of sudo dmesg which could list e.g.

[19427.230120] usb 1-2: new high-speed USB device number 2 using ehci-pci
[19427.425464] usb 1-2: config 1 interface 0 altsetting 0 bulk endpoint 0x82 has invalid maxpacket 64
[19427.425469] usb 1-2: config 1 interface 0 altsetting 0 bulk endpoint 0x3 has invalid maxpacket 64
[19427.425947] usb 1-2: New USB device found, idVendor=1ab1, idProduct=0e11, bcdDevice= 0.02
[19427.425950] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[19427.425953] usb 1-2: Product: DL3000 Serials
[19427.425955] usb 1-2: Manufacturer: Rigol Technologies. 
[19427.425957] usb 1-2: SerialNumber: DL3A204800938
[19429.525745] usbcore: registered new interface driver usbtmc

usbtmc in the last line means that the USB device has been recognized as USB Test & Measurement class device, and hence you should be able to connect to it using PyVISA as USB INSTR.

Posted by Uli Köhler in Electronics, Python

How to fix PyVISA ‘ValueError: Could not locate a VISA implementation. Install either the NI binary or pyvisa-py.’

Problem:

You are trying to use PyVISA to connect to an instrument, but you see an error message like

Traceback (most recent call last):
  File "TestPyVISA.py", line 3, in <module>
    rm = visa.ResourceManager()
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/highlevel.py", line 1526, in __new__
    visa_library = open_visa_library(visa_library)
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/highlevel.py", line 1493, in open_visa_library
    wrapper = _get_default_wrapper()
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/highlevel.py", line 1470, in _get_default_wrapper
    raise ValueError('Could not locate a VISA implementation. Install either the NI binary or pyvisa-py.')
ValueError: Could not locate a VISA implementation. Install either the NI binary or pyvisa-py.

Solution:

Install the pyvisa-py Python package:

sudo pip3 install pyvisa-py

or, if you are still using Python 2.x:

sudo pip2 install pyvisa-py

 

Posted by Uli Köhler in Python

How to fix pyaudio ‘fatal error: portaudio.h: No such file or directory’

Problem:

You are trying to install pyaudio using sudo pip install pyaudio or a similar command but you see an error message like

    src/_portaudiomodule.c:29:23: fatal error: portaudio.h: No such file or directory
     #include "portaudio.h"
                           ^
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-jgxnwixs/pyaudio/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-c3blzlv5-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-jgxnwixs/pyaudio/

Solution:

You need to install the portaudio library. On Debian/Ubuntu you can do that using

sudo apt install portaudio19-dev

on other systems either search for portaudio in your package manager or download the library from http://www.portaudio.com/

Posted by Uli Köhler in Python

Inductive reactance online calculator & Python code

Use this online calculator to compute the reactance of an inductor in Ω at a specific frequency given its inductance.

Also see Capacitive reactance online calculator & Python code

H

Hz

Formula:

X_L = 2\pi fL

Python code:

The preferred way is to use UliEngineering’s UliEngineering.Electronics.Reactance.inductive_reactance:

from UliEngineering.Electronics.Reactance import *
# You can either pass strings like "150 uH" or values like 150e-6

inductive_reactance("150 uH", "10 MHz") # returns 9424.77796076938

Or get a human-readable value:

from UliEngineering.Electronics.Reactance import *
from UliEngineering.EngineerIO import *

# Compute value as a string
xc = auto_format(inductive_reactance, "150 uH", "10 MHz") # "9.42 kΩ"

# ... or print directly
auto_print(inductive_reactance, "150 uH", "10 MHz") # prints "9.42 kΩ"

In case you can’t use UliEngineering and you want to do it manually, here’s a minimal example:

import math
def inductive_reactance(f, l):
    """Compute the inductive reactance"""
    return 2*math.pi*f*l
Posted by Uli Köhler in Calculators, Electronics, Python

Capacitive reactance online calculator & Python code

Use this online calculator to compute the reactance of a capacitor in Ω at a specific frequency given its capacitance.

Also see Inductive reactance online calculator & Python code

F

Hz

Formula:

X_C = \frac{1}{2\pi fC}

Python code:

The preferred way is to use UliEngineering’s UliEngineering.Electronics.Reactance.capacitive_reactance:

from UliEngineering.Electronics.Reactance import *
# You can either pass strings like "150 pF" or values like 150e-12

capacitive_reactance("150 pF", "10 MHz") # returns 106.1032953945969

Or get a human-readable value:

from UliEngineering.Electronics.Reactance import *
from UliEngineering.EngineerIO import *

# Compute value as a string
xc = auto_format(capacitive_reactance, "150 pF", "10 MHz") # "106 Ω"

# ... or print directly
auto_print(capacitive_reactance, "150 pF", "10 MHz") # prints "106 Ω"

In case you can’t use UliEngineering and you want to do it manually, here’s a minimal example:

import math
def capacitive_reactance(f, c):
    """Compute the capacitive reactance"""
    return 1./(2*math.pi*f*c)
Posted by Uli Köhler in Calculators, Electronics, Python

How to fix Matplotlib ‘ AttributeError: module matplotlib.pyplot’ has no attribute ‘yrange’

Problem:

You are trying to set the range of the Y axis of a matplotlib plot using code like

plt.yrange([0.0, 10.0])

but you see an error message like this:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-aa38a78ab5d7> in <module>
     12 plt.xscale('log')
     13 plt.grid(True, which="both")
---> 14 plt.yrange([0.0, 70.])
     15

AttributeError: module 'matplotlib.pyplot' has no attribute 'yrange'

Solution:

You need to use ylim since yrange does not exist! The equivalent call is:

plt.ylim([0.0, 10.0])

 

Posted by Uli Köhler in Python

How to write BytesIO content to file in Python

In order to write the contents of a BytesIO instance to a file, use this snippet:

with open("out.txt", "wb") as outfile:
    # Copy the BytesIO stream to the output file
    outfile.write(myio.getbuffer())

Note that getbuffer() will not create a copy of the values in the BytesIO buffer and will hence not consume large amounts of memory.

You can also use this function:

def write_bytesio_to_file(filename, bytesio):
    """
    Write the contents of the given BytesIO to a file.
    Creates the file or overwrites the file if it does
    not exist yet. 
    """
    with open(filename, "wb") as outfile:
        # Copy the BytesIO stream to the output file
        outfile.write(bytesio.getbuffer())

Full example:

#!/usr/bin/env python3
from io import BytesIO
import shutil

# Initialie our BytesIO
myio = BytesIO()
myio.write(b"Test 123")

def write_bytesio_to_file(filename, bytesio):
    """
    Write the contents of the given BytesIO to a file.
    Creates the file or overwrites the file if it does
    not exist yet. 
    """
    with open(filename, "wb") as outfile:
        # Copy the BytesIO stream to the output file
        outfile.write(bytesio.getbuffer())

write_bytesio_to_file("out.txt", myio)

 

Posted by Uli Köhler in Python

How to get binary data from Python’s ByteIO

Use the getvalue() function of the BytesIO instance. Example:

#!/usr/bin/env python3
from io import BytesIO

myio = BytesIO()
myio.write(b"Test 123")

print(myio.getvalue()) # Prints b'Test 123'

 

Posted by Uli Köhler in Python

How to install picamraw using pip

First try installing it normally:

sudo pip3 install picamraw

In case that fails with this error message (like for me):

Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting picamraw
Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://www.piwheels.org/simple/picamraw/

download it and install it manually: Copy the link of the most recent .whl file from https://pypi.org/project/picamraw/#files, download it using wget and install it using pip3, e.g.:

wget https://files.pythonhosted.org/packages/1e/47/4efb0d0ab5d40142424e7f3db545e276733a45bd7f7f9095919ef30c96b3/picamraw-1.2.64-py3-none-any.whl
sudo pip3 install picamraw-1.2.64-py3-none-any.whl

 

Posted by Uli Köhler in Python, Raspberry Pi

How to capture Raspi Camera image using OpenCV & Python

First, install OpenCV for Python 3:

sudo apt install python3-opencv

Here’s the code to acquire the image and store it in image.png:

#!/usr/bin/env python3
import cv2
video_capture = cv2.VideoCapture(0)
# Check success
if not video_capture.isOpened():
    raise Exception("Could not open video device")
# Read picture. ret === True on success
ret, frame = video_capture.read()

cv2.imwrite('image.png', frame)
# Close device
video_capture.release()

Run it using

python3 cv-raspicapture.py

 

Posted by Uli Köhler in OpenCV, Python, Raspberry Pi