Python

How to fix Jupyter Lab ImportError: cannot import name ‘soft_unicode’ from ‘markupsafe’

Problem:

When running

jupyter lab

you see the following error message:

Traceback (most recent call last):
  File "/usr/local/bin/jupyter-lab", line 5, in <module>
    from jupyterlab.labapp import main
  File "/usr/local/lib/python3.8/dist-packages/jupyterlab/labapp.py", line 13, in <module>
    from jupyter_server.serverapp import flags
  File "/usr/local/lib/python3.8/dist-packages/jupyter_server/serverapp.py", line 39, in <module>
    from jinja2 import Environment, FileSystemLoader
  File "/usr/lib/python3/dist-packages/jinja2/__init__.py", line 33, in <module>
    from jinja2.environment import Environment, Template
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 15, in <module>
    from jinja2 import nodes
  File "/usr/lib/python3/dist-packages/jinja2/nodes.py", line 23, in <module>
    from jinja2.utils import Markup
  File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 656, in <module>
    from markupsafe import Markup, escape, soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/usr/local/lib/python3.8/dist-packages/markupsafe/__init__.py)

Solution:

You need to install an older version of markupsafe using

sudo pip3 install markupsafe==2.0.1

until other packages have been updated.

Posted by Uli Köhler in Python

Recommended procedure for profiling python scripts

Assuming we want to profile myscript.py, run:

python3 -m cProfile -o myscript.profile myscript.py

In order to view it, I strongly recommend runsnakerun:

runsnake myscript.profile

You can install runsnake on Ubuntu using

sudo apt -y install runsnakerun

 

Posted by Uli Köhler in Python

How to set DLS ContactMe request to OpenStage 40 phones using Python

This snippet sends a ContactMe DLS provisioning request to an OpenStage 40 phone at IP 192.168.178.245 using Python. The phone will then contact 192.168.178.10 on port 18443 using HTTPS. By default (i.e. if the OpenStage 40 is not in Secure Mode, the certificate is not verified – any certificate will do!)

import requests

response = requests.post("http://192.168.178.245:8085/contact_dls.html/ContactDLS", data={
    "ContactMe": True,
    "dls_ip_addr": "192.168.178.10",
    "dls_ip_port": 18443
})
# Response will be <Response [204]> i.e. no content

Note that dls_ip_addr may also be a hostname!

Posted by Uli Köhler in Networking, Python

How to put text into input element in Pyppeteer

In Pyppeteer, if you have an input like this one:

<input id="myInput">

you can fill with text like abc123 by using page.type() like in this snippet:

await page.type('#myInput', 'abc123')

Full example

This example fetches techoverflow.net and puts my search into the search query input on the top right:

#!/usr/bin/env python3
import asyncio
from pyppeteer import launch

async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://www.techoverflow.net')
    # Fill text with input
    await page.type('.search-form-input', 'my search')
    # Make screenshot
    await page.screenshot({'path': 'screenshot.png'})
    # Cleanup
    await browser.close()

asyncio.get_event_loop().run_until_complete(main())

 

Posted by Uli Köhler in Pyppeteer, Python

How to disable SSL certificate verification in Pyppeteer

If you see an error message like

pyppeteer.errors.PageError: net::ERR_CERT_AUTHORITY_INVALID at https://10.9.5.12/

in Pyppeteer and you are sure that you just want to skip certificate verification change

browser = await launch()

to

browser = await launch({"ignoreHTTPSErrors": True})

or add "ignoreHTTPSErrors": True to the list of parameters to launch if you already have other parameters there. This will just ignore the net::ERR_CERT_AUTHORITY_INVALID and other, related HTTPS errors.

Posted by Uli Köhler in Pyppeteer, Python

Python Cloudflare DNS A record create or update example

This is based on our previous post Python Cloudflare DNS A record update example but also creates the record if it doesn’t exist.

#!/usr/bin/env python3
import CloudFlare
import argparse
import sys

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-e", "--email", required=True, help="The Cloudflare login email to use")
    parser.add_argument("-n", "--hostname", required=True, help="The hostname to update, e.g. mydyndns.mydomain.com")
    parser.add_argument("-k", "--api-key", required=True, help="The Cloudflare global API key to use. NOTE: Domain-specific API tokens will NOT work!")
    parser.add_argument("-i", "--ip-address", required=True, help="Which IP address to update the record to")
    parser.add_argument("-t", "--ttl", default=60, type=int, help="The TTL of the records in seconds (or 1 for auto)")
    args = parser.parse_args()

    # Initialize Cloudflare API client
    cf = CloudFlare.CloudFlare(
        email=args.email,
        token=args.api_key
    )
    # Get zone ID (for the domain). This is why we need the API key and the domain API token won't be sufficient
    zone = ".".join(args.hostname.split(".")[-2:]) # domain = test.mydomain.com => zone = mydomain.com
    zones = cf.zones.get(params={"name": zone})
    if len(zones) == 0:
        print(f"Could not find CloudFlare zone {zone}, please check domain {args.hostname}")
        sys.exit(2)
    zone_id = zones[0]["id"]

    # Fetch existing A record
    a_records = cf.zones.dns_records.get(zone_id, params={"name": args.hostname, "type": "A"})
    if len(a_records): # Have an existing record
        print("Found existing record, updating...")
        a_record = a_records[0]
        # Update record & save to cloudflare
        a_record["content"] = args.ip_address
        cf.zones.dns_records.put(zone_id, a_record["id"], data=a_record)
    else: # No existing record. Create !
        print("Record doesn't existing, creating new record...")
        a_record = {}
        a_record["type"] = "A"
        a_record["name"] = args.hostname
        a_record["ttl"] = args.ttl # 1 == auto
        a_record["content"] = args.ip_address
        cf.zones.dns_records.post(zone_id, data=a_record)

Usage example:

./update-dns.py --api-key ... --email [email protected] --ttl 300 --ip 1.2.3.4 --hostname mysubdomain.domain.com

 

Posted by Uli Köhler in Networking, Python

Matplotlib: How to format temperature in degrees Celsius (°C)

Based on our previous post on Matplotlib custom SI-prefix unit tick formatters, this is a simple snippet which you can use to format the Y axis of your matplotlib plots:

import matplotlib.ticker as mtick
from matplotlib import pyplot as plt

def format_celsius(value, pos=None):
    return f'{value:.1f} °C'

plt.gca().yaxis.set_major_formatter(mtick.FuncFormatter(format_celsius))

Posted by Uli Köhler in Python

How to reboot Netcup vServer using Python & SCP WSDL API

#!/usr/bin/env python3
from zeep import Client
import argparse

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-u", "--user", required=True, help="The Netcup SCP username. This is typically an integer like 92752")
    parser.add_argument("-p", "--password", required=True, help="The Netcup SCP webservice password. This is NOT the SCP login password")
    parser.add_argument("-v", "--vserver", required=True, help="The name of the vServer, like v2201261246567246578")
    args = parser.parse_args()

    client = Client('https://www.servercontrolpanel.de/WSEndUser?wsdl')

    print(client.service.vServerReset(args.user, args.password, args.vserver))

Call like this:

./restart-netcup-vserver.py --user 92752 --password su4ahK8ocu --vserver v2201261246567246578

 

Posted by Uli Köhler in Python

Python Cloudflare DNS A record update example

This script updates a DNS A record (IPv4 address) using the Cloudflare Python API. It expects the A record to be present already.

Also see Python Cloudflare DNS A record create or update example for a variant of this script which creates the record if it doesn’t exist already.

#!/usr/bin/env python3
import CloudFlare
import argparse
import sys

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-e", "--email", required=True, help="The Cloudflare login email to use")
    parser.add_argument("-n", "--hostname", required=True, help="The hostname to update, e.g. mydyndns.mydomain.com")
    parser.add_argument("-k", "--api-key", required=True, help="The Cloudflare global API key to use. NOTE: Domain-specific API tokens will NOT work!")
    parser.add_argument("-i", "--ip-address", required=True, help="Which IP address to update the record to")
    parser.add_argument("-t", "--ttl", default=60, type=int, help="The TTL of the records in seconds (or 1 for auto)")
    args = parser.parse_args()

    # Initialize Cloudflare API client
    cf = CloudFlare.CloudFlare(
        email=args.email,
        token=args.api_key
    )
    # Get zone ID (for the domain). This is why we need the API key and the domain API token won't be sufficient
    zone = ".".join(args.hostname.split(".")[-2:]) # domain = test.mydomain.com => zone = mydomain.com
    zones = cf.zones.get(params={"name": zone})
    if len(zones) == 0:
        print(f"Could not find CloudFlare zone {zone}, please check domain {args.hostname}")
        sys.exit(2)
    zone_id = zones[0]["id"]

    # Fetch existing A record
    a_record = cf.zones.dns_records.get(zone_id, params={"name": args.hostname, "type": "A"})[0]

    # Update record & save to cloudflare
    a_record["ttl"] = args.ttl # 1 == auto
    a_record["content"] = args.ip_address
    cf.zones.dns_records.put(zone_id, a_record["id"], data=a_record)

Usage example:

./update-dns.py --api-key ... --email [email protected] --ttl 300 --ip 1.2.3.4 --hostname mysubdomain.domain.com

 

Posted by Uli Köhler in Networking, Python

How to find array index closest to a given value using NumPy

Let’s say you have an 1D array like

arr = np.linspace(0, 10, 100)

and you wanted to find the array index where the value is closest to 8.5.

You can do this by first computing the absolute difference to 8.5:

np.abs(arr - 8.5)

and now using np.argmin to find the array index where the value is minimal (i.e. the index where the value of arr is closest to 8.5)

np.argmin(np.abs(arr - 8.5))

 

Posted by Uli Köhler in Python

How to create NumPy XY array from X and Y array (zip equivalent)

If you have two 1D arrays x and y of the same length n = x.shape[0] in Python, this is how you can create a shape (n, 2) array using numpy similar to the way Python’s zip does it with list objects:

xy = np.stack((x, y), axis=1)
Posted by Uli Köhler in Python

How to DC-sweep resistive voltage divider using PySpice

The following code simulates a resistive 10kΩ / 1kΩ voltage divider using PySpice using a DC sweep and can serve as a suitable starting point for simulating simple circuits. We sweep from 0V to 5V in steps of 10mV.

Also see our previous post How to simulate resistive voltage divider using PySpice for an alternate version using transient analysis:

import PySpice.Logging.Logging as Logging
logger = Logging.setup_logging()

from PySpice.Probe.Plot import plot
from PySpice.Spice.Netlist import Circuit
from PySpice.Unit import *

circuit = Circuit("MyCircuit")
# Create voltage source: 5V DC
source = circuit.VoltageSource('in', 'in', circuit.gnd, dc_value=5@u_V)
# Create resistor divider
r1 = circuit.R('R1', 'in', 'n1', 10@u_kΩ)
r2 = circuit.R('R2', 'n1', circuit.gnd, 1@u_kΩ)
# Simulate for 1 second with steps of 1 millisecond
simulator = circuit.simulator(temperature=25, nominal_temperature=25)
analysis = simulator.dc(Vin=slice(0, 5.0, 0.01))

You can access the array of output voltages of the divider (i.e. node n1) using analysis['n1']Keep in mind that if the first argument of circuit.VoltageSource is 'in', the argument to simulator.dc will be valled Vin,  not just  in! A Vwill automatically be  prepended!

 

This is the code we used to plot this:

import matplotlib.ticker as mtick
import matplotlib.pyplot as plt
from UliEngineering.EngineerIO import format_value

def format_volts(value, pos=None):
    return format_value(value, 'V')

plt.style.use("ggplot")
plt.xlabel("Input voltage")
plt.xlabel("Divider output voltage")
plt.gca().yaxis.set_major_formatter(mtick.FuncFormatter(format_volts))
plt.gca().xaxis.set_major_formatter(mtick.FuncFormatter(format_volts))
plt.gcf().set_size_inches(8,5)
plt.plot(analysis["in"], analysis["n1"], label="Input voltage")
plt.savefig("/ram/PySpice-Voltage-Divider-Sweep.svg")
Posted by Uli Köhler in Electronics, Python, SPICE

How to simulate resistive voltage divider using PySpice

The following code simulates a resistive 10kΩ / 1kΩ voltage divider using PySpice and can serve as a good starting point for simulating simple circuits.

This post shows you how to simulate the voltage divider using transient analysis. Also see an alternative variant of this post using DC sweep analysis instead: How to DC-sweep resistive voltage divider using PySpice

import PySpice.Logging.Logging as Logging
logger = Logging.setup_logging()

from PySpice.Probe.Plot import plot
from PySpice.Spice.Netlist import Circuit
from PySpice.Unit import *

circuit = Circuit("MyCircuit")
# Create voltage source: 5V DC
source = circuit.VoltageSource('V1', 'in', circuit.gnd, dc_value=5@u_V)
# Create resistor divider
r1 = circuit.R('R1', 'in', 'n1', 10@u_kΩ)
r2 = circuit.R('R2', 'n1', circuit.gnd, 1@u_kΩ)
# Simulate for 1 second with steps of 1 millisecond
simulator = circuit.simulator(temperature=25, nominal_temperature=25)
analysis = simulator.transient(step_time=1@u_ms, end_time=1@u_s)

You can access the array of output voltages of the divider (i.e. node n1) using analysis['n1']:

This is the code we used to plot this:

import matplotlib.ticker as mtick
import matplotlib.pyplot as plt
from UliEngineering.EngineerIO import format_value

def format_volts(value, pos=None):
    return format_value(value, 'V')

plt.style.use("ggplot")
plt.xlabel("Time [ms]")
plt.ylim([0.0, 5.5])
plt.gca().yaxis.set_major_formatter(mtick.FuncFormatter(format_volts))
plt.gcf().set_size_inches(8,5)
plt.plot(analysis["in"], label="Input voltage")
plt.plot(analysis["n1"], label="Voltage divider output")
plt.gca().legend()
plt.savefig("/ram/PySpice-Voltage-Divider.svg")

 

Posted by Uli Köhler in Electronics, Python, SPICE

How to create resistor in PySpice: Minimal example

This creates a resistor named R1 which has its + pin connected to node n1 and its - pin connected to GND. The value of the resistor is set to 1kΩ.

r1 = circuit.R('R1', 'n1', circuit.gnd, 1@u_kΩ)

Full example:

import PySpice.Logging.Logging as Logging
logger = Logging.setup_logging()

from PySpice.Probe.Plot import plot
from PySpice.Spice.Netlist import Circuit
from PySpice.Unit import *

circuit = Circuit("MyCircuit")
r1 = circuit.R('R1', 'n1', circuit.gnd, 1@u_kΩ)

 

Posted by Uli Köhler in Electronics, Python, SPICE

How to create constant voltage source in PySpice

This creates a voltage source named V1 which has its + pin connected to node n1 and its - pin connected to GND. The voltage source is set to 5V DC.

source = circuit.VoltageSource('V1', 'n1', circuit.gnd, dc_value=5@u_V)

Full example:

import PySpice.Logging.Logging as Logging
logger = Logging.setup_logging()

from PySpice.Probe.Plot import plot
from PySpice.Spice.Netlist import Circuit
from PySpice.Unit import *

circuit = Circuit("MyCircuit")
source = circuit.VoltageSource('V1', 'n1', circuit.gnd, dc_value=5@u_V)

 

Posted by Uli Köhler in Electronics, Python, SPICE

How to replace numpy NaNs by last non-NaN value

Use bottleneck.push:

import bottleneck

y = bottleneck.push(y, axis=0)

If not installed already, you can install bottleneck using

pip install bottleneck

 

Posted by Uli Köhler in Python

How to set matplotlib X axis timestamp format to hours and minutes

import matplotlib.dates as mdates

plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%H:%M'))

 

Posted by Uli Köhler in Python

How to change matplotlib X axis timestamp format

Sometimes you get weird timestamp format choices in Matplotlib such as 22 19:40 (meaning 19:40 o’clock on the 22nd of whatever month).

This is easy to fix:

import matplotlib.dates as mdates

plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%H:%M'))

which will result in this image, for example:

Posted by Uli Köhler in Python

LinuxCNC: How to get current position including offset using Python

In our previous example LinuxCNC: How to find current position using Python we showed how to use stat.actual_position to get the current position in machine coordinates using LinuxCNC’s Python API.

#!/usr/bin/env python2.7
import linuxcnc

stat = linuxcnc.stat()
stat.poll()
x,y,z,a,b,c,u,v,w = stat.actual_position

# NOTE: Ignore ABCUVW since not used for my machine
# Subtract G5x offset
xo,yo,zo,ao,bo,co,uo,vo,wo = stat.g5x_offset

x -= xo
y -= yo
z -= zo

# Subtract tool offset
xo,yo,zo,ao,bo,co,uo,vo,wo = stat.tool_offset

x -= xo
y -= yo
z -= zo

# Print offset coordinates
print(x,y,z)

 

Posted by Uli Köhler in LinuxCNC, Python

LinuxCNC: What attributes can you read from a linuxcnc.stat() object?

As shown in our previous example LinuxCNC: How to find current position using Python you can get an

These are the attributes of the object returned by linuxcnc.stat() for LinuxCNC 2.7:

acceleration
active_queue
actual_position
adaptive_feed_enabled
ain
angular_units
aout
axes
axis
axis_mask
block_delete
call_level
command
current_line
current_vel
cycle_time
debug
delay_left
din
distance_to_go
dout
dtg
echo_serial_number
enabled
estop
exec_state
feed_hold_enabled
feed_override_enabled
feedrate
file
flood
g5x_index
g5x_offset
g92_offset
gcodes
homed
id
inpos
input_timeout
interp_state
interpreter_errcode
joint_actual_position
joint_position
kinematics_type
limit
linear_units
lube
lube_level
max_acceleration
max_velocity
mcodes
mist
motion_line
motion_mode
motion_type
optional_stop
paused
pocket_prepped
poll
position
probe_tripped
probe_val
probed_position
probing
program_units
queue
queue_full
queued_mdi_commands
rapidrate
read_line
rotation_xy
settings
spindle_brake
spindle_direction
spindle_enabled
spindle_increasing
spindle_override_enabled
spindle_speed
spindlerate
state
task_mode
task_paused
task_state
tool_in_spindle
tool_offset
tool_table
velocity

You can find those out using

#!/usr/bin/env python2.7
import linuxcnc

stat = linuxcnc.stat()
stat.poll()
for entry in dir(stat):
    print(entry)

 

Posted by Uli Köhler in LinuxCNC, Python