Python

Oct

2019

How to get nanoseconds from pandas.Timedelta object

If you have a pandas.Timedelta object, you can use Timedelta.total_seconds() to get the seconds as a floating-point number with nanosecond resolution and then multiply with one billion (1e9, the number of nanoseconds in one second) to obtain the number of nanoseconds in the Timedelta:

timedelta.total_seconds() * 1e9

In case you want an integer, use

int(round(timedelta.total_seconds() * 1e9))

Note that using round() is required here to avoid errors due to floating point precision.

or use this function definition:

def nanoseconds_from_timedelta(timedelta):
    """Compute the nanoseconds in a timedelta as floating-point number"""
    return timedelta.total_seconds() * 1e9

def nanoseconds_from_timedelta_integer(timedelta):
    """Compute the nanoseconds in a timedelta as integer number"""
    return int(round(timedelta.total_seconds() * 1e9))

# Usage example:
ns = nanoseconds_from_timedelta(timedelta)
print(ns) # Prints 2000751999.9999998

ns = nanoseconds_from_timedelta_integer(timedelta)
print(ns) # Prints 2000752000

Oct

2019

How to create pandas.Timedelta object from two timestamps

If you have two pandas.Timestamp objects, you can simply substract them using the minus operator (-) in order to obtain a pandas.Timedelta object:

import pandas as pd
import time

# Create two timestamps
ts1 = pd.Timestamp.now()
time.sleep(2)
ts2 = pd.Timestamp.now()

# The difference of these timestamps is a pandas.Timedelta object.
timedelta = ts2 - ts1
print(timedelta) # Prints '0 days 00:00:02.000752'

Oct

2019

How to get current timestamp in Pandas

Use

import pandas as pd

current_timestamp = pd.Timestamp.now()

pd.Timestamp.now() will return the timestamp as pandas.Timestamp object.

Example:

>>> import pandas as pd
>>> pd.Timestamp.now()
Timestamp('2019-10-27 16:11:43.998993')

Oct

2019

How to fix ModuleNotFoundError: No module named ‘dateutil’

Problem:

You want to use the dateutil library in Python, but when you try to import it, you see an exception like this:

Traceback (most recent call last):
  File "my.py", line 11, in <module>
    import dateutil as dl
ModuleNotFoundError: No module named 'dateutil'

When you try installing it using e.g. pip install dateutil you see this error message:

Collecting dateutil
  Could not find a version that satisfies the requirement dateutil (from versions: )
No matching distribution found for dateutil

Solution:

The package is named python-dateutil. Install using

sudo pip install python-dateutil

or (for Python3)

sudo pip3 install python-dateutil

Oct

2019

How to get schema of SQLite3 table in Python

Also see How to show table schema for SQLite3 table on the command line

Use this function to find the table schema of a SQLite3 table in Python:

def sqlite_table_schema(conn, name):
    """Return a string representing the table's CREATE"""
    cursor = conn.execute("SELECT sql FROM sqlite_master WHERE name=?;", [name])
    sql = cursor.fetchone()[0]
    cursor.close()
    return sql

Usage example:

print(sqlite_table_schema(conn, 'commands'))

Full example:

#!/usr/bin/env python3
import sqlite3
conn = sqlite3.connect('/usr/share/command-not-found/commands.db')

def sqlite_table_schema(conn, name):
    cursor = conn.execute("SELECT sql FROM sqlite_master WHERE name=?;", [name])
    sql = cursor.fetchone()[0]
    cursor.close()
    return sql

print(sqlite_table_schema(conn, 'commands'))

which prints

CREATE TABLE "commands" 
           (
            [cmdID] INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
            [pkgID] INTEGER NOT NULL,
            [command] TEXT,
            FOREIGN KEY ([pkgID]) REFERENCES "pkgs" ([pkgID])
           )

Oct

2019

How to fix SQLite3 Python ‘Incorrect number of bindings supplied. The current statement uses 1, … supplied’

Problem:

You are trying to run a simple SQL query with placeholders on a SQLite3 database, e.g.:

name = "mytable"
conn.execute("SELECT sql FROM sqlite_master WHERE name=?;", name)

But you get an exception like this:

---------------------------------------------------------------------------
ProgrammingError                          Traceback (most recent call last)
<ipython-input-55-e385cf40fd72> in <module>
      1 name = "mytable"
----> 2 conn.execute("SELECT sql FROM sqlite_master WHERE type='table' AND name=?;", name)

ProgrammingError: Incorrect number of bindings supplied. The current statement uses 1, and there are 7 supplied.

Solution:

You need to use a list as the second argument to conn.execute(...)!

Since you only gave the function a string, the string is being interpreted as list of characters.

In our example from above, you just need to wrap name in square brackets to read [name]:

name = "mytable"
conn.execute("SELECT sql FROM sqlite_master WHERE name=?;", [name])

Oct

2019

How to list tables in SQLite3 database in Python

Also see How to list SQLite3 database tables on command line

You can use this snippet to list all the SQL tables in your SQLite 3.x database in Python:

def tables_in_sqlite_db(conn):
    cursor = conn.execute("SELECT name FROM sqlite_master WHERE type='table';")
    tables = [
        v[0] for v in cursor.fetchall()
        if v[0] != "sqlite_sequence"
    ]
    cursor.close()
    return tables

Usage example:

#!/usr/bin/env python3
import sqlite3
# Open database
conn = sqlite3.connect('/usr/share/command-not-found/commands.db')
# List tables
tables = tables_in_sqlite_db(conn)

# Your code goes here!
# Example:
print(tables) # prints ['commands', 'packages']

Aug

2019

How to intercept AJAX JSON response in Pyppeteer

This example shows you how to intercept and print the content of a JSON response requested via any AJAX request on a web page by using Pyppeteer:

import asyncio
import json
from pyppeteer import launch

async def intercept_network_response(response):
    # In this example, we care only about responses returning JSONs
    if "application/json" in response.headers.get("content-type", ""):
        # Print some info about the responses
        print("URL:", response.url)
        print("Method:", response.request.method)
        print("Response headers:", response.headers)
        print("Request Headers:", response.request.headers)
        print("Response status:", response.status)
        # Print the content of the response
        try:
            # await response.json() returns the response as Python object
            print("Content: ", await response.json())
        except json.decoder.JSONDecodeError:
            # NOTE: Use await response.text() if you want to get raw response text
            print("Failed to decode JSON from", await response.text())

async def main():
    browser = await launch()
    page = await browser.newPage()
    
    page.on('response', intercept_network_response)
            
    await page.goto('https://instagram.com')
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

Aug

2019

Pyppeteer minimal network response interception example

Using Javascript (puppeteer)? Check out Minimal puppeteer response interception example

This example shows you how to intercept network responses in pyppeteer.

Note: This intercepts the response, not the request! This means you can abort the request before it is actually sent to the server, but you can’t read the content of the response! See Pyppetteer minimal network request interception example for an example on how to intercept requests.

import asyncio
from pyppeteer import launch

async def intercept_network_response(response):
    # In this example, we only care about HTML responses!
    if "text/html" in response.headers.get("content-type", ""):
        # Print some info about the responses
        print("URL:", response.url)
        print("Method:", response.request.method)
        print("Response headers:", response.headers)
        print("Request Headers:", response.request.headers)
        print("Response status:", response.status)
        # Print the content of the response
        print("Content: ", await response.text())
        # NOTE: Use await response.json() if you want to get the JSON directly

async def main():
    browser = await launch()
    page = await browser.newPage()
    
    page.on('response', intercept_network_response)
            
    await page.goto('https://techoverflow.net')
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

Aug

2019

How to fix Python NameError: name ‘reponse’ is not defined

If you see this error message in Python:

NameError: name 'reponse' is not defined

you likely just mis-spelled response !

reponse  <-- your code, you're missing an 's'!
response <-- correct

Aug

2019

Pyppeteer minimal network request interception example

Using Javascript (puppeteer)? Check out Minimal puppeteer request interception example

This example shows you how to intercept network requests in pyppeteer:

Note: This intercepts the request, not the response! This means you can abort the request made, but you can’t read the content of the response! See Pyppetteer minimal network response interception example for an example on how to intercept responses.

import asyncio
from pyppeteer import launch

async def intercept_network_request(request):
    # Print some info about the request
    print("URL:", request.url)
    print("Method:", request.method)
    print("Headers:", request.headers)
    # NOTE: You can also await request.abort() to abort the requst1
    await request.continue_()

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.setRequestInterception(True)
    
    page.on('request', intercept_network_request)
            
    await page.goto('https://techoverflow.net')
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

Aug

2019

How to fix pyppeteer.errors.NetworkError: Request interception is not enabled.

Note: Also see Pyppetteer minimal network request interception example

Problem:

You are trying to intercept a request in Pyppeteer using

page.on('request', my_intercept_request)

but you’re getting an error message like this:

Traceback (most recent call last):
  File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/dist-packages/pyee/_compat.py", line 62, in _callback
    self.emit('error', exc)
  File "/usr/local/lib/python3.6/dist-packages/pyee/_base.py", line 106, in emit
    self._emit_handle_potential_error(event, args[0] if args else None)
  File "/usr/local/lib/python3.6/dist-packages/pyee/_base.py", line 83, in _emit_handle_potential_error
    raise error
  File "run.py", line 6, in intercept
    await request.continue_()
  File "/usr/local/lib/python3.6/dist-packages/pyppeteer/network_manager.py", line 481, in continue_
    raise NetworkError('Request interception is not enabled.')
pyppeteer.errors.NetworkError: Request interception is not enabled.

Solution:

Add

await page.setRequestInterception(True)

directly after your call to

page = await browser.newPage()

This will enable request interception and your code will run just fine.

Aug

2019

How to fix PyVISA “No module named ‘serial.tools'”

Problem:

You want to use an ASRL (serial) instrument in PyVISA, but when you run

python3 -m visa info

you get this output even though you have serial installed:

ASRL INSTR:
   Please install PySerial (>=3.0) to use this resource type.
   No module named 'serial.tools'

Solution:

You have installed serial but you need to install pyserial – they are not the same!

First you need to remove the system package python3-serial if installed. Example for Ubuntu/Debian:

sudo apt remove python3-serial

and also remove the pip serial package if installed

sudo pip3 uninstall serial

Then install pyserial:

sudo pip3 install pyserial

You can check if PySerial is installed properly using

python3 -m visa info

It should show you

ASRL INSTR: Available via PySerial (3.4)

once pyserial is installed correctly!

Note: The commands above are for Python 3.x. In case you are still using Python 2.x use pip2 or pip instead of pip3 and use python-serial instead of python3-serial as APT package name.

Aug

2019

How to fix PyVISA not finding any ASRL (serial port) instruments

Problem:

You are trying to connect to a USB instrument using PyVISA & pyvisa-py, but the PyVISA resource manager doesn’t find any instruments:

#!/usr/bin/env python3
import visa
rm = visa.ResourceManager()
print(rm.list_resources()) # Prints "()" => No instruments found!

Solution:

Install PySerial 3.0+:

First you need to remove the system package python3-serial if installed. Example for Ubuntu/Debian:

sudo apt remove python3-serial

and also remove the pip serial package if installed (we need to install pyserial, not serial!)

sudo pip3 uninstall serial

Then install pyserial:

sudo pip3 install pyserial

You can check if PySerial is installed properly using

python3 -m visa info

It should show you

ASRL INSTR: Available via PySerial (3.4)

if pyserial is installed correctly!

Note: The commands above are for Python 3.x. In case you are still using Python 2.x use pip2 or pip instead of pip3 and use python-serial instead of python3-serial as APT package name.

Aug

2019

How to fix PyVISA ‘Found a device whose serial number cannot be read. The partial VISA resource name is: USB0::[…]::[…]::???::0::INSTR’

Problem:

You are trying to list available resources using PyVISA e.g. using

import visa
rm = visa.ResourceManager()
print(rm.list_resources())

But when you try to run it, you see an output like

Found a device whose serial number cannot be read. The partial VISA resource name is: USB0::6833::3601::???::0::INSTR
()

Solution:

Even though PyVISA doesn’t tell you that exactly, this is just the bog-standard Linux USB permission problem. We already provided a generic solution in How to fix ALL USB permission issues on Linux once and for all.

Excerpt from this post (see there for details on why it works):

Run this in your favourite shell:

wget https://techoverflow.net/scripts/udev-install-usbusers.sh | sudo bash -s $USER

This will print:

SUBSYSTEM=="usb", MODE="0666", GROUP="usbusers"
USB device configuration has been installed. Please log out and log back in or reboot

then log out and log back in (or close your SSH session and log back in).

In case this doesn’t work, reboot!

After that, your PyVISA script should work as intended and should print e.g.

('USB0::6833::3601::DL3A204800938::0::INSTR')

Aug

2019

How to fix PyVISA not finding any USB instruments

Problem:

You are trying to connect to a USB instrument using PyVISA & pyvisa-py, but the PyVISA resource manager doesn’t find any instruments:

#!/usr/bin/env python3
import visa
rm = visa.ResourceManager()
print(rm.list_resources()) # Prints "()" => No instruments found!

Solution:

In order for pyvisa-py to be able to connect to USB instruments, you need to install the Python usb library!

On Debian or Ubuntu, install it using

sudo apt-get -y install python3-usb

or, if you are still using Python 2.x

sudo apt-get -y install python-usb

Now, re-run the script – you should see an output like

('USB0::6833::3601::DL3A204800938::0::INSTR',)

In case you still don’t see the output, run python3 -m visa info or python -m visa info (for Python 2.x).

It should show an output like this:

Machine Details:
   Platform ID:    Linux-4.19.0-5-686-i686-with-debian-10.0
   Processor:      

Python:
   Implementation: CPython
   Executable:     /usr/bin/python3
   Version:        3.7.3
   Compiler:       GCC 8.3.0
   Bits:           32bit
   Build:          Apr  3 2019 05:39:12 (#default)
   Unicode:        UCS4

PyVISA Version: 1.9.1

Backends:
   ni:
      Version: 1.9.1 (bundled with PyVISA)
      Binary library: Not found
   py:
      Version: 0.3.1
      ASRL INSTR: Available via PySerial (3.4)
      USB INSTR: Available via PyUSB (1.0.2). Backend: libusb1
      USB RAW: Available via PyUSB (1.0.2). Backend: libusb1
      TCPIP INSTR: Available 
      TCPIP SOCKET: Available 
      GPIB INSTR:
         Please install linux-gpib to use this resource type.
         No module named 'gpib'

Check Backends -> py -> USB INSTR: If it’s not Available via PyUSB, check the information message for hints what might be the issue. For example, if it says

USB INSTR:
   Please install PyUSB to use this resource type.
   No module named 'usb'

that means that the Python USB library has not been installed properly.

If USB is Available via PyUSB but PyVISA still doesn’t find the instrument, check if it is connected properly using

lsusb

which should show a line related to your instrument’s manufacturer, e.g.

Bus 001 Device 002: ID 1ab1:0e11 Rigol Technologies

Also unplug and re-plug your instrument so Linux tries to reconnect to the USB device and check the end of the output of sudo dmesg which could list e.g.

[19427.230120] usb 1-2: new high-speed USB device number 2 using ehci-pci
[19427.425464] usb 1-2: config 1 interface 0 altsetting 0 bulk endpoint 0x82 has invalid maxpacket 64
[19427.425469] usb 1-2: config 1 interface 0 altsetting 0 bulk endpoint 0x3 has invalid maxpacket 64
[19427.425947] usb 1-2: New USB device found, idVendor=1ab1, idProduct=0e11, bcdDevice= 0.02
[19427.425950] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[19427.425953] usb 1-2: Product: DL3000 Serials
[19427.425955] usb 1-2: Manufacturer: Rigol Technologies. 
[19427.425957] usb 1-2: SerialNumber: DL3A204800938
[19429.525745] usbcore: registered new interface driver usbtmc

usbtmc in the last line means that the USB device has been recognized as USB Test & Measurement class device, and hence you should be able to connect to it using PyVISA as USB INSTR.

Aug

2019

How to fix PyVISA ‘ValueError: Could not locate a VISA implementation. Install either the NI binary or pyvisa-py.’

Problem:

You are trying to use PyVISA to connect to an instrument, but you see an error message like

Traceback (most recent call last):
  File "TestPyVISA.py", line 3, in <module>
    rm = visa.ResourceManager()
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/highlevel.py", line 1526, in __new__
    visa_library = open_visa_library(visa_library)
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/highlevel.py", line 1493, in open_visa_library
    wrapper = _get_default_wrapper()
  File "/usr/local/lib/python3.7/dist-packages/pyvisa/highlevel.py", line 1470, in _get_default_wrapper
    raise ValueError('Could not locate a VISA implementation. Install either the NI binary or pyvisa-py.')
ValueError: Could not locate a VISA implementation. Install either the NI binary or pyvisa-py.

Solution:

Install the pyvisa-py Python package:

sudo pip3 install pyvisa-py

or, if you are still using Python 2.x:

sudo pip2 install pyvisa-py

Aug

2019

How to fix pyaudio ‘fatal error: portaudio.h: No such file or directory’

Problem:

You are trying to install pyaudio using sudo pip install pyaudio or a similar command but you see an error message like

    src/_portaudiomodule.c:29:23: fatal error: portaudio.h: No such file or directory
     #include "portaudio.h"
                           ^
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-jgxnwixs/pyaudio/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-c3blzlv5-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-jgxnwixs/pyaudio/

Solution:

You need to install the portaudio library. On Debian/Ubuntu you can do that using

sudo apt install portaudio19-dev

on other systems either search for portaudio in your package manager or download the library from http://www.portaudio.com/

Jul

2019

How to create a self-deleting temporary directory in Python

Starting from Python 3.2 your can use tmpfile.TemporaryDirectory like this:

from tempfile import TemporaryDirectory

with TemporaryDirectory(prefix="myapp-") as tmpdir:
    print(tmpdir) # e.g. "/tmp/myapp-fevhzh93"

# Once you are outside the "with" block, the directory will be deleted!

In case you are stuck with using older Python versions, check out How to create temporary directory in Python.

Jul

2019

Inductive reactance online calculator & Python code

Use this online calculator to compute the reactance of an inductor in Ω at a specific frequency given its inductance.

Also see Capacitive reactance online calculator & Python code

TechOverflow calculators:
You can enter values with SI suffixes like 12.2m (equivalent to 0.012) or 14k (14000) or 32u (0.000032).
The results are calculated while you type and shown directly below the calculator, so there is no need to press return or click on a Calculate button. Just make sure that all inputs are green by entering valid values.

H ⚠

Hz ⚠

Formula:

X_L = 2\pi fL

Python code:

The preferred way is to use UliEngineering’s UliEngineering.Electronics.Reactance.inductive_reactance:

from UliEngineering.Electronics.Reactance import *
# You can either pass strings like "150 uH" or values like 150e-6

inductive_reactance("150 uH", "10 MHz") # returns 9424.77796076938

Or get a human-readable value:

from UliEngineering.Electronics.Reactance import *
from UliEngineering.EngineerIO import *

# Compute value as a string
xc = auto_format(inductive_reactance, "150 uH", "10 MHz") # "9.42 kΩ"

# ... or print directly
auto_print(inductive_reactance, "150 uH", "10 MHz") # prints "9.42 kΩ"

In case you can’t use UliEngineering and you want to do it manually, here’s a minimal example:

import math
def inductive_reactance(f, l):
    """Compute the inductive reactance"""
    return 2*math.pi*f*l

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Formula:

Python code:

Search

Categories