Python

How to fix KiBot ‘KiBoM not installed or too old’

Problem:

When running kibot-check, you see the following warning message:

* KiBoM not installed or too old
  Visit: https://github.com/INTI-CMNB/KiBoM
  Download it from: https://github.com/INTI-CMNB/KiBoM/releases
  - Mandatory for `kibom`

but even installing kibom using pip install kibom, the warning does not disappear.

Solution:

You need to install a specific fork of KiBom:

pip install git+https://github.com/INTI-CMNB/KiBoM

After that, the warning will disappear.

Posted by Uli Köhler in Electronics, KiCAD, Python

How to compute & plot sun path diagram using skyfield in Python

In this example, we’ll show how to generate a sun path diagram for any given day for a preselected location. By using the skyfield library, we can obtain extremely accurate predictions of the sun’s position in the sky, and in contrast to many other libraries, we can use the same method to predict the position of other celestial bodies.

#!/usr/bin/env python3
from skyfield import api
from skyfield import almanac
from datetime import datetime
from datetime import timedelta
from matplotlib import pyplot as plt
import pytz
import dateutil.parser
from collections import namedtuple
from UliEngineering.Utils.Date import *
import matplotlib.ticker as mtick

# Matplotlib formatter
def format_degrees(value, pos=None):
    return f'{value:.0f} °'

ts = api.load.timescale()
ephem = api.load_file('de421.bsp')

sun = ephem["Sun"]
earth = ephem["Earth"]

# Data types
AltAz = namedtuple("AltAz", ["altitude", "azimuth", "distance"])
TimeAltAz = namedtuple("TimeAltAz", ["time", "altitude", "azimuth", "distance"])

def sun_altaz(planet_at_location, ts):
    # Compute the sun position as seen from the observer at <location>
    sun_pos = planet_at_location.at(ts).observe(sun).apparent()
    # Compute apparent altitude & azimuth for the sun's position
    altitude, azimuth, distance = sun_pos.altaz()
    return AltAz(altitude, azimuth, distance)

def sun_altaz_for_day(location, year, month, day, tz):
    earth_at_location = (earth + location)
    
    minutes = list(yield_minutes_on_day(year=year, month=month, day=day, tz=tz))
    skyfield_minutes = ts.from_datetimes(minutes)

    minutely_altaz = [sun_altaz(earth_at_location, ts) for ts in skyfield_minutes]
    # Extract components for plotting
    minutely_alt = [altaz.altitude.degrees for altaz in minutely_altaz]
    minutely_az = [altaz.azimuth.degrees for altaz in minutely_altaz]
    minutely_distance = [altaz.distance for altaz in minutely_altaz]

    return TimeAltAz(minutes, minutely_alt, minutely_az, minutely_distance)

# Random location near Munich
location = api.Topos('48.324777 N', '11.405610 E', elevation_m=519)
# Compute Alt/Az for two different days
jun = sun_altaz_for_day(location, 2022, 6, 15, pytz.timezone("Europe/Berlin"))
dec = sun_altaz_for_day(location, 2022, 12, 15, pytz.timezone("Europe/Berlin"))

# Plot!
plt.gca().yaxis.set_major_formatter(mtick.FuncFormatter(format_degrees))
plt.gca().xaxis.set_major_formatter(mtick.FuncFormatter(format_degrees))

plt.plot(jun.azimuth, jun.altitude, label="15th of June")
plt.plot(dec.azimuth, dec.altitude, label="15th of December")
plt.ylim([0, None]) # Do not show sun angles below the horizon
plt.ylabel("Sun altitude")
plt.xlabel("Sun azimuth")
plt.title("Apparent sun position at our office")
plt.grid() 
plt.gca().legend()
plt.gcf().set_size_inches(10,5)

plt.savefig("Sun path.svg")

 

Posted by Uli Köhler in Physics, Python, skyfield

How to generate datetime for every hour on a given day in Python

This example code generates a timezone-aware datetime for every hour on a given day (minutes & seconds are always set to 0) in a given timezone.

First, install the UliEngineering library and pytz for timezones:

pip install --user UliEngineering pytz

Now you can use UliEngineering.Utils.Date.yield_hours_on_day():

from UliEngineering.Utils.Date import *

for hour in yield_hours_on_day(year=2022, month=6, day=15, tz=pytz.timezone("Europe/Berlin"):
    pass # TODO: Your code goes here

Or, if you want to have a list of datetime instances instead of a generator:

from UliEngineering.Utils.Date import *

hours = list(yield_hours_on_day(year=2022, month=6, day=15, tz=pytz.timezone("Europe/Berlin")))
Posted by Uli Köhler in Python

How to generate datetime for every second on a given day in Python

This example code generates a timezone-aware datetime for every second on a given day in a given timezone.

First, install the UliEngineering library and pytz for timezones:

pip install --user UliEngineering pytz

Now you can use UliEngineering.Utils.Date.yield_seconds_on_day():

from UliEngineering.Utils.Date import *

for second in yield_seconds_on_day(year=2022, month=6, day=15, tz=pytz.timezone("Europe/Berlin"):
    pass # TODO: Your code goes here

Or, if you want to have a list of datetime instances instead of a generator:

from UliEngineering.Utils.Date import *

seconds = list(yield_seconds_on_day(year=2022, month=6, day=15, tz=pytz.timezone("Europe/Berlin")))
Posted by Uli Köhler in Python

How to generate datetime for every minute on a given day in Python

This example code generates a timezone-aware datetime for every minute (the seconds are always set to 0) for a given day in a given timezone.

First, install the UliEngineering library and pytz for timezones:

pip install --user UliEngineering pytz

Now you can use UliEngineering.Utils.Date.yield_minutes_on_day():

from UliEngineering.Utils.Date import *

for minute in yield_minutes_on_day(year=2022, month=6, day=15, tz=pytz.timezone("Europe/Berlin"):
    pass # TODO: Your code goes here

Or, if you want to have a list of datetime instances instead of a generator:

from UliEngineering.Utils.Date import *

minutes = list(yield_minutes_on_day(year=2022, month=6, day=15, tz=pytz.timezone("Europe/Berlin")))
Posted by Uli Köhler in Python

Matplotlib: How to format angle in degrees (°)

Based on our previous post on Matplotlib custom SI-prefix unit tick formatters, this is a simple snippet which you can use to format the Y axis of your matplotlib plots. In our example, the function shows 2 digits after the decimal points (.2f) but you can change that to how ever many you prefer.

import matplotlib.ticker as mtick
from matplotlib import pyplot as plt

def format_degrees(value, pos=None):
    return f'{value:.2f} °'

plt.gca().yaxis.set_major_formatter(mtick.FuncFormatter(format_degrees))

Example diagram

From our post How to compute & plot sun path diagram using skyfield in Python

Posted by Uli Köhler in Python

How to convert skyfield Time into datetime at specific timezone

When you have a skyfield Time object like

t = ts.now()
# Example: <Time tt=2459750.027604357>

you can convert it to a Python datetime in a specific timezone (Europe/Berlin in this example) using .astimezone() and the pytz library:

t.astimezone(pytz.timezone("Europe/Berlin"))
# Example: datetime.datetime(2022, 6, 19, 14, 38, 35, 832445, tzinfo=<DstTzInfo 'Europe/Berlin' CEST+2:00:00 DST>)

Complete example

from skyfield import api
from datetime import datetime
import pytz

ts = api.load.timescale()
t = ts.now()
dt = t.astimezone(pytz.timezone("Europe/Berlin"))
print(dt) # e.g. 2022-06-19 14:42:47.406786+02:00

 

Posted by Uli Köhler in Python, skyfield

How to compute position of sun in the sky in Python using skyfield

The following code computes the position of the sun (in azimuth/altitude coordinates) in the sky using the skyfield library. Note that you need to download de421.bsp .

from skyfield import api
from skyfield import almanac
from datetime import datetime
from datetime import timedelta
import dateutil.parser
from calendar import monthrange

ts = api.load.timescale()
ephem = api.load_file('de421.bsp')

sun = ephem["Sun"]
earth = ephem["Earth"]

# Compute sunrise & sunset for random location near Munich
location = api.Topos('48.324777 N', '11.405610 E', elevation_m=519)
# Compute the sun position as seen from the observer at <location>
sun_pos = (earth + location).at(ts.now()).observe(sun).apparent()
# Compute apparent altitude & azimuth for the sun's position
altitude, azimuth, distance = sun_pos.altaz()

# Print results (example)
print(f"Altitude: {altitude.degrees:.4f} °")
print(f"Azimuth: {azimuth.degrees:.4f} °")

Example output

Altitude: -3.3121 °
Azimuth: 48.4141 °

Factors influencing the accuracy of the calculation

This way of calculating the position takes into account:

  • The slight shift in position caused by light speed
  • The very very slight shift in position caused by earth’s gravity

But it does not take into account:

  • Atmospheric distortions shifting the sun’s position
  • The extent of the sun’s disk causing the sun to emanate not from a point but apparently from an area
Posted by Uli Köhler in Physics, Python, skyfield

How to fix Python skyfield FileNotFoundError: [Errno 2] No such file or directory: ‘de421.bsp’

Problem:

When trying to use the Python skyfield library, you see an exception like

Input In [2], in <cell line: 11>()
      8 from calendar import monthrange
     10 ts = api.load.timescale()
---> 11 ephem = api.load_file('de413.bsp')

File /usr/local/lib/python3.10/dist-packages/skyfield/iokit.py:412, in load_file(path)
    410 base, ext = os.path.splitext(path)
    411 if ext == '.bsp':
--> 412     return SpiceKernel(path)
    413 raise ValueError('unrecognized file extension: {}'.format(path))

File /usr/local/lib/python3.10/dist-packages/skyfield/jpllib.py:71, in SpiceKernel.__init__(self, path)
     69 self.path = path
     70 self.filename = os.path.basename(path)
---> 71 self.spk = SPK.open(path)
     72 self.segments = [SPICESegment(self, s) for s in self.spk.segments]
     73 self.codes = set(s.center for s in self.segments).union(
     74                  s.target for s in self.segments)

File /usr/local/lib/python3.10/dist-packages/jplephem/spk.py:49, in SPK.open(cls, path)
     46 @classmethod
     47 def open(cls, path):
     48     """Open the file at `path` and return an SPK instance."""
---> 49     return cls(DAF(open(path, 'rb')))

FileNotFoundError: [Errno 2] No such file or directory: 'de421.bsp'

Solution:

Take a look at the api.load(...) line in your code:

ephem = api.load_file('de421.bsp')

It tries to load the data from the file de421.bsp in the current directory. This file contains positional data of objects in the sky and you need to manually download that file.

You can download the file from NASA. Just take care to either place it into the right directory or modifying the path in the api.load() call to point to the file.

URL for downloading the file:

https://ssd.jpl.nasa.gov/ftp/eph/planets/bsp/de421.bsp

My preferred way to download it is using wget:

wget https://ssd.jpl.nasa.gov/ftp/eph/planets/bsp/de421.bsp

This command will place the file into the current directory.

Posted by Uli Köhler in Python, skyfield

How to compute sunrise & sunset in Python using skyfield

The following code will compute the sunrise & sunset at a specific location & elevation using the skyfield library. Note that you need to download de413.bsp .

from skyfield import api
from skyfield import almanac
from datetime import datetime
from datetime import timedelta
import dateutil.parser
from calendar import monthrange

ts = api.load.timescale()
ephem = api.load_file('de413.bsp')

def compute_sunrise_sunset(location, year=2019, month=1, day=1):
    t0 = ts.utc(year, month, day, 0)
    # t1 = t0 plus one day
    t1 = ts.utc(t0.utc_datetime() + timedelta(days=1))
    t, y = almanac.find_discrete(t0, t1, almanac.sunrise_sunset(ephem, location))
    sunrise = None
    for time, is_sunrise in zip(t, y):
        if is_sunrise:
            sunrise = dateutil.parser.parse(time.utc_iso())
        else:
            sunset = dateutil.parser.parse(time.utc_iso())
    return sunrise, sunset

# Compute sunrise & sunset for random location near Munich
location = api.Topos('48.324777 N', '11.405610 E', elevation_m=519)
now = datetime.now()
sunrise, sunset = compute_sunrise_sunset(location, now.year, now.month, now.day)

# Print result (example)
print(f'Sunrise today: {sunrise}')
print(f'Sunset today: {sunset}')

Definition of sunrise & sunset in this context

According to the skyfield documentation:

Skyfield uses the same definition as the United States Naval Observatory: the Sun is up when its center is 0.8333 degrees below the horizon, which accounts for both its apparent radius of around 16 arcminutes and also for the 34 arcminutes by which atmospheric refraction on average lifts the image of the Sun.

Other caveats

  • Note that obstructions like mountains are not taken into account for this model
  • Note that the resulting timestamps are UTC, if you want local time, you’ll have to convert them appropriately

Example output:

Sunrise today: 2022-06-19 03:12:56+00:00
Sunset today: 2022-06-19 19:18:38+00:00

 

Posted by Uli Köhler in Physics, Python, skyfield

How to fix Python skyfield FileNotFoundError: [Errno 2] No such file or directory: ‘de413.bsp’

Problem:

When trying to use the Python skyfield library, you see an exception like

Input In [2], in <cell line: 11>()
      8 from calendar import monthrange
     10 ts = api.load.timescale()
---> 11 ephem = api.load_file('de413.bsp')

File /usr/local/lib/python3.10/dist-packages/skyfield/iokit.py:412, in load_file(path)
    410 base, ext = os.path.splitext(path)
    411 if ext == '.bsp':
--> 412     return SpiceKernel(path)
    413 raise ValueError('unrecognized file extension: {}'.format(path))

File /usr/local/lib/python3.10/dist-packages/skyfield/jpllib.py:71, in SpiceKernel.__init__(self, path)
     69 self.path = path
     70 self.filename = os.path.basename(path)
---> 71 self.spk = SPK.open(path)
     72 self.segments = [SPICESegment(self, s) for s in self.spk.segments]
     73 self.codes = set(s.center for s in self.segments).union(
     74                  s.target for s in self.segments)

File /usr/local/lib/python3.10/dist-packages/jplephem/spk.py:49, in SPK.open(cls, path)
     46 @classmethod
     47 def open(cls, path):
     48     """Open the file at `path` and return an SPK instance."""
---> 49     return cls(DAF(open(path, 'rb')))

FileNotFoundError: [Errno 2] No such file or directory: 'de413.bsp'

Solution:

Take a look at the api.load(...) line in your code:

ephem = api.load_file('de413.bsp')

It tries to load the data from the file de413.bsp in the current directory. This file contains positional data of objects in the sky and you need to manually download that file.

You can download the file from NASA. Just take care to either place it into the right directory or modifying the path in the api.load() call to point to the file.

URL for downloading the file:

https://ssd.jpl.nasa.gov/ftp/eph/planets/bsp/de413.bsp

My preferred way to download it is using wget:

wget https://ssd.jpl.nasa.gov/ftp/eph/planets/bsp/de413.bsp

This command will place the file into the current directory.

Posted by Uli Köhler in Python, skyfield

How to install ESP32 esptool / esptool.py on Ubuntu

The recommended way is to install the current version using pip since the version installed using apt might be out of date:

sudo pip install esptool

After that, you can use esptool.py – note that you need to call it as esptool.py, not just as esptool!

In case you are missing pip , install python3-dev using apt:

sudo apt -y install python3-pip

 

Posted by Uli Köhler in ESP8266/ESP32, Linux, Python

How to convert collections.Counter to pandas DataFrame

Pandas can take care of the conversion of a Counter to a DataFrameby itself but you need to add a column label:

pd.DataFrame({"YourColumnLabelGoesHere": counterObject})

Full example

import pandas as pd
from collections import Counter

ctr = Counter()
ctr["a"] += 1
ctr["b"] += 1
ctr["a"] += 1
ctr["a"] += 1
ctr["b"] += 1
ctr["a"] += 1
ctr["c"] += 1

pd.DataFrame({"ctr": ctr})

This will result in the following DataFrame:

 

Posted by Uli Köhler in pandas, Python

structlog minimal example

import structlog

logger = structlog.get_logger()

# Usage example
logger.info("Test log")

 

Posted by Uli Köhler in Python

How to fix matplotlib OSError: ‘xkcd’ not found in the style library

Problem:

While trying to enable the matplotlib xkcd style using

plt.style.use("xkcd")

you see the following error message:

OSError: 'xkcd' not found in the style library and input is not a valid URL or path; see `style.available` for list of available styles

Solution:

You can’t enable xkcd-style plots by running plt.style.use("xkcd"). Instead, use with plt.xkcd():

with plt.xkcd():
    # TODO your plotting code goes here!
    # plt.plot(x, y) # Example

 

Posted by Uli Köhler in Python

Recommended library for executing shell commands in Python

I recommend using invoke instead of the built-in subprocess to handle executing any shell command in Python.

Not only does it provide a more user friendly syntax compared to e.g. subprocess.check_output():

run('make')

but it also tends to act more like you’d expect especially regarding the output of the command and has easy-to-use parameters such as hide=True to hide the output of shell commands.

Furthermore, it provides a buch of really useful features such as automatically responding to prompts from the shell command.

Posted by Uli Köhler in Python

How to fix Python pyarrow pip install error: Could NOT find Arrow (missing: Arrow_DIR)

Problem:

When trying to install pyarrow such as using

pip install pyarrow

you see an error log like

      -- Found Python3Alt: /home/uli/.pypy3-virtualenv/bin/pypy3
      CMake Warning (dev) at /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:273 (message):
        The package name passed to `find_package_handle_standard_args` (PkgConfig)
        does not match the name of the calling package (Arrow).  This can lead to
        problems in calling code that expects `find_package` result variables
        (e.g., `_FOUND`) to follow a certain pattern.
      Call Stack (most recent call first):
        /usr/share/cmake-3.18/Modules/FindPkgConfig.cmake:59 (find_package_handle_standard_args)
        cmake_modules/FindArrow.cmake:39 (include)
        cmake_modules/FindArrowPython.cmake:46 (find_package)
        CMakeLists.txt:229 (find_package)
      This warning is for project developers.  Use -Wno-dev to suppress it.
      
      -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.2")
      -- Could NOT find Arrow (missing: Arrow_DIR)
      -- Checking for module 'arrow'
      --   No package 'arrow' found
      CMake Error at /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message):
        Could NOT find Arrow (missing: ARROW_INCLUDE_DIR ARROW_LIB_DIR
        ARROW_FULL_SO_VERSION ARROW_SO_VERSION)
      Call Stack (most recent call first):
        /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 (_FPHSA_FAILURE_MESSAGE)
        cmake_modules/FindArrow.cmake:450 (find_package_handle_standard_args)
        cmake_modules/FindArrowPython.cmake:46 (find_package)
        CMakeLists.txt:229 (find_package)
      
      
      -- Configuring incomplete, errors occurred!
      See also "/tmp/pip-install-409dctif/pyarrow_b70cde6894c3469483f7360493fc2e65/build/temp.linux-x86_64-pypy39/CMakeFiles/CMakeOutput.log".
      error: command '/usr/bin/cmake' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects

Solution:

You need to install the arrow library in order to be able to compile pyarrow from source. On Ubuntu, this can be done using

sudo apt install -y -V ca-certificates lsb-release wget
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb -O /tmp/apache-arrow.deb
sudo apt -y install /tmp/apache-arrow.deb
sudo apt -y update
sudo apt -y install libarrow-dev libarrow-python-dev

 

Posted by Uli Köhler in Python

How to fix Python Pillow pip install exception: RequiredDependencyException: jpeg

Problem:

When trying to install pillow such as using

pip install Pillow

you see an error log like

      running build_ext
      
      
      The headers or library files could not be found for jpeg,
      a required dependency when compiling Pillow from source.
      
      Please see the install instructions at:
         https://pillow.readthedocs.io/en/latest/installation.html
      
      Traceback (most recent call last):
        File "/tmp/pip-install-_g5fa7ox/pillow_7cb18c0d6bec468e8844184b98c8bf45/setup.py", line 989, in <module>
          setup(
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/_distutils/core.py", line 148, in setup
          return run_commands(dist)
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
          dist.run_commands()
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
          self.run_command(cmd)
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/dist.py", line 1214, in run_command
          super().run_command(command)
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/command/install.py", line 68, in run
          return orig.install.run(self)
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/_distutils/command/install.py", line 670, in run
          self.run_command('build')
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/dist.py", line 1214, in run_command
          super().run_command(command)
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/_distutils/command/build.py", line 136, in run
          self.run_command(cmd_name)
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/dist.py", line 1214, in run_command
          super().run_command(command)
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/command/build_ext.py", line 79, in run
          _build_ext.run(self)
        File "/home/uli/.pypy3-virtualenv/lib/pypy3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 339, in run
          self.build_extensions()
        File "/tmp/pip-install-_g5fa7ox/pillow_7cb18c0d6bec468e8844184b98c8bf45/setup.py", line 804, in build_extensions
          raise RequiredDependencyException(f)
      RequiredDependencyException: jpeg
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-_g5fa7ox/pillow_7cb18c0d6bec468e8844184b98c8bf45/setup.py", line 1009, in <module>
          raise RequiredDependencyException(msg)
      RequiredDependencyException:
      
      The headers or library files could not be found for jpeg,
      a required dependency when compiling Pillow from source.
      
      Please see the install instructions at:
         https://pillow.readthedocs.io/en/latest/installation.html
      
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> Pillow

Solution:

Pillow needs a buch of libraries to be installed in order to work properly. Use the following command from the official Pillow website on Ubuntu:

sudo apt-get install cmake libtiff5-dev libjpeg8-dev libopenjp2-7-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python3-tk libharfbuzz-dev libfribidi-dev libxcb1-dev

or check out the installation guide for commands for other operating systems.

Posted by Uli Köhler in Python

How to fix Python MongoDB TypeError: Object of type ObjectId is not JSON serializable

Problem:

When trying to export data as JSON that has originally been queried from MongoDB using code like

with open("alle.json", "w") as outfile:
    json.dump(alle, outfile)

you see the following error message:

File /usr/lib/python3.9/json/__init__.py:179, in dump(obj, fp, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    173     iterable = cls(skipkeys=skipkeys, ensure_ascii=ensure_ascii,
    174         check_circular=check_circular, allow_nan=allow_nan, indent=indent,
    175         separators=separators,
    176         default=default, sort_keys=sort_keys, **kw).iterencode(obj)
    177 # could accelerate with writelines in some versions of Python, at
    178 # a debuggability cost
--> 179 for chunk in iterable:
    180     fp.write(chunk)

File /usr/lib/python3.9/json/encoder.py:429, in _make_iterencode.<locals>._iterencode(o, _current_indent_level)
    427     yield _floatstr(o)
    428 elif isinstance(o, (list, tuple)):
--> 429     yield from _iterencode_list(o, _current_indent_level)
    430 elif isinstance(o, dict):
    431     yield from _iterencode_dict(o, _current_indent_level)

File /usr/lib/python3.9/json/encoder.py:325, in _make_iterencode.<locals>._iterencode_list(lst, _current_indent_level)
    323         else:
    324             chunks = _iterencode(value, _current_indent_level)
--> 325         yield from chunks
    326 if newline_indent is not None:
    327     _current_indent_level -= 1

File /usr/lib/python3.9/json/encoder.py:405, in _make_iterencode.<locals>._iterencode_dict(dct, _current_indent_level)
    403         else:
    404             chunks = _iterencode(value, _current_indent_level)
--> 405         yield from chunks
    406 if newline_indent is not None:
    407     _current_indent_level -= 1

File /usr/lib/python3.9/json/encoder.py:438, in _make_iterencode.<locals>._iterencode(o, _current_indent_level)
    436         raise ValueError("Circular reference detected")
    437     markers[markerid] = o
--> 438 o = _default(o)
    439 yield from _iterencode(o, _current_indent_level)
    440 if markers is not None:

File /usr/lib/python3.9/json/encoder.py:179, in JSONEncoder.default(self, o)
    160 def default(self, o):
    161     """Implement this method in a subclass such that it returns
    162     a serializable object for ``o``, or calls the base implementation
    163     (to raise a ``TypeError``).
   (...)
    177 
    178     """
--> 179     raise TypeError(f'Object of type {o.__class__.__name__} '
    180                     f'is not JSON serializable')

TypeError: Object of type ObjectId is not JSON serializable

Solution:

This error occurs because objects queried from PyMongo always contain _id which is of type ObjectId and the normal JSON library (or drop-in replacements like simplejson do not know how to create JSON representations of Objects of type ObjectId).

In order to fix this, use pymongo‘s json_util instead of json. Note that the bson.json_util package contains dumps but does not contain dump, so use the following snippet to write to a file:

 

import bson.json_util as json_util

with open("alle.json", "w") as outfile:
    outfile.write(json_util.dumps(alle))

 

Posted by Uli Köhler in MongoDB, Python

How to iterate all databases in PyMongo

This short example shows how to iterate all databases or list all database names for a MongoDB in Python using pymongo:

from pymongo import MongoClient

client = MongoClient("mongodb://localhost")

for database in client.list_databases():
    print(database['name'])

s

Posted by Uli Köhler in Python