Python

How to parse filename extension from URL using Python

If you have an URL like

https://logodix.com/logo/1667872.jpg?param=value

and you want to find just the filename extension (.jpg for this example), use the following code:

from urllib.parse import urlsplit
import os.path

url = "https://logodix.com/logo/1667872.jpg?param=value"

path = urlsplit(url).path
extension = os.path.splitext(path)[-1] # e.g. ".jpg"

 

Posted by Uli Köhler in Python

How to fix Python 3 AttributeError: ‘function’ object has no attribute ‘urlsplit’

Problem:

When trying to urlsplit an URL in Python 3:

from urllib.parse import urlparse

path = urlparse.urlsplit(remote_image_url).path

you see the following error message:

AttributeError                            Traceback (most recent call last)
Input In [30], in <cell line: 3>()
      1 from urllib.parse import urlparse
----> 3 path = urlparse.urlsplit(remote_image_url).path
      4 filename = posixpath.basename(path)

AttributeError: 'function' object has no attribute 'urlsplit'

Solution:

The equivalent of urlparse.urlsplit() in Python 3 is urllib.parse.urlsplit().

Therefore, a working code example is

from urllib.parse import urlsplit

path = urlsplit(remote_image_url).path

 

Posted by Uli Köhler in Python

How to fix Python 3 ModuleNotFoundError: No module named ‘urlparse’

Problem:

While trying to import urlparse in Python 3 using

import urlparse

you see the following error message:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Input In [29], in <cell line: 1>()
----> 1 import urlparse

ModuleNotFoundError: No module named 'urlparse'

Solution:

In Python 3, urlparse has been renamed to urllib.urlparse. In order to import it, use this line instead:

from urllib.parse import urlparse

 

Posted by Uli Köhler in Python

How to auto-restart bottle server when Python file changes using nodemon

Assuming you have a Python script server.py that you want to auto-reload every time the file changes, use the following script using nodemon:

nodemon -w . --exec python server.py

The -w . tells  nodemon files to watch for changes of all files in the current directory (.)

I generally recommend creating a script start.sh to automatically run this command:

#!/bin/sh
nodemon -w . --exec python server.py

and then make it executable using

chmod a+x start.sh

Now you can run it using

./start.sh

 

Posted by Uli Köhler in Python

How to return JSON array using bottle – minimal example

Returning a JSON array in bottle involves setting the response content type to application/json and then manually dumping the array using json.dumps():

#!/usr/bin/env python3
from bottle import route, run, response
import json

@route('/')
def json_array():
    response.content_type = 'application/json'
    return json.dumps(["a", "b", "c"])

run(host='localhost', port=9000)

 

Posted by Uli Köhler in Python

How to auto-restart bottle server when Python file changes using entr

Note: I do not recommend using entr any more since for me it doesn’t work reliably. I recommend using nodemon instead, see How to auto-restart bottle server when Python file changes using nodemon

Assuming you have a Python script server.py that you want to auto-reload every time the file changes, use the following script using entr:

ls *.py | entr python server.py

The ls *.py tells entr which files to watch for changes.

Posted by Uli Köhler in Python

Bottle HTTP server with dedicated server class – minimal example

This script uses a dedicated class MyServer to encapsulate the bottle server appropriately. Compared to our previous post Python bottle minimal example it provides better encapsulation of the server at the cost of slightly more lines of code.

#!/usr/bin/env python3
from bottle import Bottle, run

class MyServer(object):
    def __init__(self):
        self.app = Bottle()
        self.init_routes()
        
    def init_routes(self):
        """Initialize all routes"""
        @self.app.route('/hello')
        def hello():
            return "Hello World!"

    def run(self):
        run(self.app, host='0.0.0.0', port=8080)
        
# Example usage
if __name__ == "__main__":
    server = MyServer()
    server.run()

How to use

Run the server

python bottle-server.py

and open

http://localhost:8080/hello

Now you should see the Hello World! message.

Posted by Uli Köhler in Python

Inventree Python API: How to list all stock locations

See our previous post Inventree Python API: Minimal API connect example using YAML config for our method of creating the api object using a YAML config file!

from inventree.stock import StockLocation

all_stock_locations = StockLocation.list(api)

# Dict of part categories by name
# (e.g. 'OpAmps')
stock_locations_by_name = {
    category["name"]: category
    for category in all_stock_locations
}
# Dict of part categories by public key (e.g. 7)
part_locations_by_pk = {
    category.pk: category
    for category in all_stock_locations
}
# Dict of part categories by hierarchical path
# (e.g. 'Office/Spart parts box')
stock_locations_by_pathstring = {
    category.pathstring: category
    for category in all_stock_locations
}

 

Posted by Uli Köhler in InvenTree, Python

Inventree Python API: How to create a new part

See our previous post Inventree Python API: Minimal API connect example using YAML config for our method of creating the api object using a YAML config file!

The following code finds a part category called ICs and creates a new part in that category.

First, we need to fetch the part categories, this is from our previous post Inventree Python API: Minimal API connect example using YAML config:

from inventree.part import PartCategory

all_categories = PartCategory.list(api)

# Dict of part categories by name
# (e.g. 'OpAmps')
part_categories_by_name = {
    category["name"]: category
    for category in all_categories
}
# Dict of part categories by public key (e.g. 7)
part_categories_by_pk = {
    category.pk: category
    for category in all_categories
}
# Dict of part categories by hierarchical path
# (e.g. 'Electronics-Components/ICs/OpAmps')
part_categories_by_pathstring = {
    category.pathstring: category
    for category in all_categories
}

Now let’s select the correct category:

ics = part_categories_by_name['ICs']

Now it’s finally time to create the part:

from inventree.part import Part

new_part = Part.create(api, {
    'name': 'L78L33ABD',
    'description': '100mA 3.3V fixed LDO regulator, SOIC-8',
    'category': ics.pk
})

You can read the primary key (pk) for the newly generated part using

new_part.pk
Posted by Uli Köhler in InvenTree, Python

Inventree Python API: How to list all part categories

See our previous post Inventree Python API: Minimal API connect example using YAML config for our method of creating the api object using a YAML config file!

from inventree.part import PartCategory

all_categories = PartCategory.list(api)

# Dict of part categories by name
# (e.g. 'OpAmps')
part_categories_by_name = {
    category["name"]: category
    for category in all_categories
}
# Dict of part categories by public key (e.g. 7)
part_categories_by_pk = {
    category.pk: category
    for category in all_categories
}
# Dict of part categories by hierarchical path
# (e.g. 'Electronics-Components/ICs/OpAmps')
part_categories_by_pathstring = {
    category.pathstring: category
    for category in all_categories
}

Note that this code by itself does not take into account the hierarchy of the categories.

Example output

part_categories_by_name:

{'Elektronik-Komponenten': <inventree.part.PartCategory at 0x7fd735125510>,
 'ICs': <inventree.part.PartCategory at 0x7fd7356d1720>,
 'OpAmps': <inventree.part.PartCategory at 0x7fd735ead4e0>}

part_categories_by_pk:

{1: <inventree.part.PartCategory at 0x7fd7346b5db0>,
 2: <inventree.part.PartCategory at 0x7fd73438aa10>,
 3: <inventree.part.PartCategory at 0x7fd727325b40>}

part_categories_by_pathstring:

{'Elektronik-Komponenten': <inventree.part.PartCategory at 0x7fd727bd3040>,
 'Elektronik-Komponenten/ICs': <inventree.part.PartCategory at 0x7fd727bd1db0>,
 'Elektronik-Komponenten/ICs/OpAmps': <inventree.part.PartCategory at 0x7fd727bd1060>}

 

Posted by Uli Köhler in InvenTree, Python

Inventree Python API: Minimal API connect example using YAML config

This example allows you to create a config file inventree.yml:

server: 'https://inventree.mydomain.com'
username: 'admin'
password: 'Chuiquu5Oqu8il5ahLoja4aecai4ee'

and then read it in Python and connect to the Inventree API:

from inventree.api import InvenTreeAPI

# Load config
import yaml

with open("inventree.yml", "r") as file:
    config = yaml.safe_load(file)
    
api = InvenTreeAPI(config["server"], username=config["username"], password=config["password"])

 

Posted by Uli Köhler in InvenTree, Python

PySerial minimal RFC2217 example: Copy data received from serial port to stdout

Also see the same example for a local serial port: PySerial minimal example: Copy data received from serial port to stdout

This example connects to the RFC2217 remote serial port on 10.1.2.3 port 1234. It does not send data to the serial port but only copies data received from the serial port to stdout.

#!/usr/bin/env python3
import serial
with serial.serial_for_url("rfc2217://10.1.2.3:1234", baudrate=115200) as ser:
    try:
        while True:
            response = ser.read()
            if response:
                print(response.decode("iso-8859-1"), end="")
    finally:
        ser.close()

 

By using iso-8859-1   decoding, we ensure that even binary bytes are decoded in some way and do not cause an exception.

Posted by Uli Köhler in Embedded, Python

How to download forum pages to HTML using Python & requests

This will download page 1000 … page 1 in descending order and save the HTML to 1000.html up to 1.html

import requests
for i in range(1000, 0, -1):
    with open(f"{i}.html", "w") as outfile:
        outfile.write(requests.get(f"https://forum.my-domain.com/showthread.php?123456-my-thread/page{i}").text)

With progress bar

import requests
from tqdm import tqdm
for i in tqdm(range(1000, 0, -1)):
    with open(f"{i}.html", "w") as outfile:
        outfile.write(requests.get(f"https://forum.my-domain.com/showthread.php?123456-my-thread/page{i}").text)

 

Posted by Uli Köhler in Python

How to setup & use WordPress REST API authentication using Python

WordPress authentication plugin setup

First, install the WordPress REST API Authentication wordpress plugin, which you can find by searching for WordPress REST API Authentication:

Then you need to open the plugin configuration page. Open Plugins in the WordPress admin panel, locate the  WordPress REST API Authentication plugin and click Configure

Select Basic Authentication:

Then click Next on the top right:

and click Finish on the next page:

Setup in Python

Assuming you have a WordPress user admin with password abc123 we can modify our code from How to get WordPress posts as JSON using Python & the WordPress REST API in order to query a non-public endpoint:

import requests
import base64

# Compute basic authentication header
auth_header = b"Basic " + base64.b64encode(b"admin:abc123")

# posts is a list of JSON objects, each representing a post
posts = requests.get("https://mydomain.com/wp-json/wp/v2/posts",
                     params={"context": "edit"},
                     headers={"Authorization": auth_header}).json()

 

Posted by Uli Köhler in Python, Wordpress

How to get all WordPress posts as JSON using Python & the WordPress REST API

In our previous post How to get WordPress posts as JSON using Python & the WordPress REST API we showed how to fetch a single page of 10 posts using the WordPress REST API in Python.

In this post, we’ll use the pagination in order to fetch a list of all the posts.

Firstly, we observe that once we query an invalid page such as ?page=1000000, the returned JSON will be

{'code': 'rest_post_invalid_page_number',
 'message': 'The page number requested is larger than the number of pages available.',
 'data': {'status': 400}}

instead of the JSON array representing the list of posts.

Using this information, we can write a fetcher that fetches pages of 100 posts each until this error message is encountered:

from tqdm import tqdm
import requests

def page_numbers():
    """Infinite generate of page numbers"""
    num = 1
    while True:
        yield num
        num += 1

posts = []
for page in tqdm(page_numbers()):
    # Fetch the next [pagesize=10] posts
    posts_page = requests.get("https://mydomain.com/wp-json/wp/v2/posts", params={"page": page, "per_page": 100}).json()
    # Check for "last page" error code
    if isinstance(posts_page, dict) and posts_page["code"] == "rest_post_invalid_page_number": # Found last page
        break
    # No error code -> add posts
    posts += posts_page

 

 

Posted by Uli Köhler in Python, Wordpress

How to get WordPress posts as JSON using Python & the WordPress REST API

You can use the requests library to fetch the posts as JSON from /wp-json/wp/v2/posts

On the wordpress site, you typically don’t need to configure this – the JSON API is enabled by default and accessing publically available info can be done without authentication.

import requests

# posts is a list of JSON objects, each representing a post
posts = requests.get("https://mydomain.com/wp-json/wp/v2/posts").json()

This will, by default, fetch the most recent 10 posts. See TODO for more info on how to fetch all posts using pagination

Posted by Uli Köhler in Python, Wordpress

Netmiko MikroTik RouterOS minimal example

This example prints the identity (i.e. user-defined name) of the switch/router at IP address 10.0.0.1 with password abc123abc.

from netmiko import ConnectHandler
mikrotik = {
    'device_type': 'mikrotik_routeros',
    'host':   '10.0.0.1',
    'username': 'admin',
    'password': 'abc123abc'
}

mikrotik_connection = ConnectHandler(**mikrotik)
print(mikrotik_connect.send_command(f'/system/identity/print', cmd_verify=False))

Example output:

name: MySwitch01

 

Posted by Uli Köhler in MikroTik, Networking, Python

How to find first element in sorted numpy array that is larger than a given scalar

Also see How to find first element in sorted numpy array that is smaller than a given scalar

You can use np.searchsorted() like this:

idx = np.searchsorted(arr, scalar, side='right') + 1

Full example:

import numpy as np

# Example array
arr = np.array([1, 2, 3, 4, 5])

# We'll search for this value
scalar = 3.5

# Use numpy.searchsorted() to find the first element in the array that is larger than the scalar
# We need to use + 1 since searchsorted() returns the first element that is smaller than the scalar
idx = np.searchsorted(arr, scalar, side='right') + 1

# Print the resulting index
print(idx)

 

Posted by Uli Köhler in Python

How to find first element in sorted numpy array that is smaller than a given scalar

Also see How to find first element in sorted numpy array that is larger than a given scalar

You can use np.searchsorted() like this:

idx = np.searchsorted(arr, scalar, side='left')

Full example:

import numpy as np

# Example array
arr = np.array([1, 2, 3, 4, 5])

# We'll search for this value
scalar = 3.5

# Use numpy.searchsorted() to find the first element in the array that is smaller than the scalar
idx = np.searchsorted(arr, scalar, side='left')

# Print the resulting index
print(idx)

 

Posted by Uli Köhler in Python