How to write BytesIO content to file in Python

In order to write the contents of a BytesIO instance to a file, use this snippet:

with open("out.txt", "wb") as outfile:
    # Copy the BytesIO stream to the output file
    outfile.write(myio.getbuffer())

Note that getbuffer() will not create a copy of the values in the BytesIO buffer and will hence not consume large amounts of memory.

You can also use this function:

def write_bytesio_to_file(filename, bytesio):
    """
    Write the contents of the given BytesIO to a file.
    Creates the file or overwrites the file if it does
    not exist yet. 
    """
    with open(filename, "wb") as outfile:
        # Copy the BytesIO stream to the output file
        outfile.write(bytesio.getbuffer())

Full example:

#!/usr/bin/env python3
from io import BytesIO
import shutil

# Initialie our BytesIO
myio = BytesIO()
myio.write(b"Test 123")

def write_bytesio_to_file(filename, bytesio):
    """
    Write the contents of the given BytesIO to a file.
    Creates the file or overwrites the file if it does
    not exist yet. 
    """
    with open(filename, "wb") as outfile:
        # Copy the BytesIO stream to the output file
        outfile.write(bytesio.getbuffer())

write_bytesio_to_file("out.txt", myio)

 

Posted by Uli Köhler in Python

How to get binary data from Python’s ByteIO

Use the getvalue() function of the BytesIO instance. Example:

#!/usr/bin/env python3
from io import BytesIO

myio = BytesIO()
myio.write(b"Test 123")

print(myio.getvalue()) # Prints b'Test 123'

 

Posted by Uli Köhler in Python

How to install picamraw using pip

First try installing it normally:

sudo pip3 install picamraw

In case that fails with this error message (like for me):

Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting picamraw
Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://www.piwheels.org/simple/picamraw/

download it and install it manually: Copy the link of the most recent .whl file from https://pypi.org/project/picamraw/#files, download it using wget and install it using pip3, e.g.:

wget https://files.pythonhosted.org/packages/1e/47/4efb0d0ab5d40142424e7f3db545e276733a45bd7f7f9095919ef30c96b3/picamraw-1.2.64-py3-none-any.whl
sudo pip3 install picamraw-1.2.64-py3-none-any.whl

 

Posted by Uli Köhler in Python, Raspberry Pi

How to capture Raspi Camera image using OpenCV & Python

First, install OpenCV for Python 3:

sudo apt install python3-opencv

Here’s the code to acquire the image and store it in image.png:

#!/usr/bin/env python3
import cv2
video_capture = cv2.VideoCapture(0)
# Check success
if not video_capture.isOpened():
    raise Exception("Could not open video device")
# Read picture. ret === True on success
ret, frame = video_capture.read()

cv2.imwrite('image.png', frame)
# Close device
video_capture.release()

Run it using

python3 cv-raspicapture.py

 

Posted by Uli Köhler in OpenCV, Python, Raspberry Pi

How to fix Raspi camera ‘mmal_vc_component_create: failed to create component ‘vc.ril.camera’ (1:ENOMEM)’

Problem:

You are trying to acess the Raspberry Pi camera using raspistill or raspivid, but you see an error message like this:

mmal: Cannot read camera info, keeping the defaults for OV5647
mmal: mmal_vc_component_create: failed to create component 'vc.ril.camera' (1:ENOMEM)
mmal: mmal_component_create_core: could not create component 'vc.ril.camera' (1)
mmal: Failed to create camera component
mmal: main: Failed to create camera component
mmal: Camera is not enabled in this build. Try running "sudo raspi-config" and ensure that "camera" has been enabled

Solution:

The most common issue here is that you don’t have the camera interface enabled. Read How to enable Raspberry Pi camera using raspi-config for instructions on how to do that.

If the camera interface has already been enabled (remember that you need to reboot for the changes to take effect), the most likely reason for error messages like this is that the camera is not connected correctly to the Raspberry Pi:

  • Is the CSI cable inserted the right way? The silvery contacts need to face away from the Ethernet connector!
  • Is the CSI cable fully seated?
  • Did you insert the CSI cable into the Display connector? It needs to be inserted into the CSI connector, which is the one closer to the Ethernet connector.
  • Is the other end of the CSI cable correctly attached to the camera board?
Posted by Uli Köhler in Embedded, Raspberry Pi

How to enable Raspberry Pi camera using raspi-config

You can enable the camera interface by running

sudo raspi-config

Select 5 Interfacing Options and press Return.

Now select P1 Camera and press Return.

Select Yes and press Return.

Now that the camera interface has been enabled, press Return.

Press Tab twice to select Finish and press Return. You are now asked if you want to reboot:

Select Yes and wait for your Raspberry Pi to reboot. Your camera interface will be enabled after the reboot.

In case you are not asked to reboot, your camera interface was already enabled. In this case, select Finish on the main raspi-config screen by pressing Tab twice and pressing Return:

After pressing Return, check your shell again. Reboot (sudo reboot) so that any previous settings will take effect and then try

sudo raspistill -o myimg.jpg

in order to test your camera.

Posted by Uli Köhler in Embedded, Raspberry Pi

How to fix Raspi camera ‘failed to open vchiq instance’

Problem:

You are trying to access the Raspberry Pi camera using raspistill or raspivid. Although you already enabled the camera interface using raspi-config, you see this error message:

* failed to open vchiq instance

Solution:

Your user does not have the permissions to access the camera interface. As a quick fix, you can run raspistill or raspivid as root using sudo, or add your user to the video group to acquire the required permissions:

sudo usermod -a -G video $USER

After doing this, log out and log back in again for the changes to take effect – or close your SSH session and connect again. If in doubt, try to reboot.

Posted by Uli Köhler in Embedded, Raspberry Pi

How to add a "Press enter to close this window" message to your Python script

Just add this line at the bottom of your Python script:

input("Press Enter to close this window...")

Python 2 variant:

raw_input("Press Enter to close this window...")

 

Posted by Uli Köhler in Python

nginx: Where to get $request_basename from

nginx by default does not provide a $request_basename, but you can define it yourself using a regex named group

location ~ ^/(?P<request_basename>[^/]+)$ {
    # TODO: Add your code here - you can use $request_basename!
}

For a usage example, see my previous post on nginx: Force file download using Content-Disposition for entire directory.

Posted by Uli Köhler in nginx

How to extract only filename (basename) in nginx

You can easily extract only the filename (e.g. myfile.txt for https://domain.com/directory/myfile.txt) by capturing it in a regex named group:

location ~ ^/(?P<request_basename>[^/]+)$ {
    # TODO: Add your code here - you can use $request_basename!
}

For a usage example, see my previous post on nginx: Force file download using Content-Disposition for entire directory.

Posted by Uli Köhler in nginx

nginx: Force file download using Content-Disposition for entire directory

To force downloading all files in a directory using nginx by using the Content-Disposition header, use this snippet:

# Force files in /scripts to download instead of display!
location ~ ^/scripts/(?P<request_basename>[^/]+)$ {
    add_header Content-Disposition "attachment; filename=$request_basename";
}

Note that this might cause issues with some mobile iOS devices which do not use a proper filesystem.

Original source: coderwall.com (but heavily modified)

Posted by Uli Köhler in nginx

How to check if your filesystem is mounted in noatime, relatime or strictatime mode

If you need to use a software that depends on your filesystem storing the last access time of a file (atime), you can use this script to check if your filesystem is mounted in noatime, strictatime or relatime mode.

This script works on both Linux and Windows.

On Linux, you can simply run this

wget -qO- https://techoverflow.net/scripts/check-atime.py | python3

Python 2 version (pythonclock.org !)

wget -qO- https://techoverflow.net/scripts/check-atime.py | python

Note that the script will check for the atime mode in whichever directory you run the script in.

On Windows, download the script and directly open it using Python. In case you don’t have Python installed, install it from the Microsoft store or download it here before downloading the script.

In case you need to check the atime mode of a specific drive (C:, D:, …), download, the script, place it in that directory and run it from there.

This script will print one of three messages:

  • Your filesystem is mounted in NOATIME mode – access times will NEVER be updated automatically
  • Your filesystem is mounted in RELATIME mode – access times will only be updated if they are too old
  • Your filesystem is mounted in STRICTATIME mode – access times will be updated on EVERY file access

On Linux, the default is relatime whereas on Windows the default is strictatime.

Sourcecode of the script:

#!/usr/bin/env python3
"""
This utility script checks which atime mode (strictatime, relatime or noatime)
is in use for the current filesystem
"""
import os
import time
from datetime import datetime

def datetime_to_timestamp(dt):
    return time.mktime(dt.timetuple()) + dt.microsecond/1e6

def set_file_access_time(filename, atime):
    """
    Set the access time of a given filename to the given atime.
    atime must be a datetime object.
    """
    stat = os.stat(filename)
    mtime = stat.st_mtime
    os.utime(filename, (datetime_to_timestamp(atime), mtime))


def last_file_access_time(filename):
    """
    Get a datetime() representing the last access time of the given file.
    The returned datetime object is in local time
    """
    return datetime.fromtimestamp(os.stat(filename).st_atime)

try:
    # Create test file
    with open("test.txt", "w") as outfile:
        outfile.write("test!")
    time.sleep(0.1)
    # Read & get first atime
    with open("test.txt") as infile:
        infile.read()
    atime1 = last_file_access_time("test.txt")
    # Now read file
    time.sleep(0.1)
    with open("test.txt") as infile:
        infile.read()
    # Different atime after read?
    atime2 = last_file_access_time("test.txt")
    # Set OLD atime for relatime check!
    set_file_access_time("test.txt", datetime(2000, 1, 1, 0, 0, 0))
    # Access again
    with open("test.txt") as infile:
        infile.read()
    # Different atime now
    atime3 = last_file_access_time("test.txt")
    # Check atime
    changed_after_simple_access = atime2 > atime1
    changed_after_old_atime = atime3 > atime1
    # Convert mode to text and print
    if (not changed_after_simple_access) and (not changed_after_old_atime):
        print("Your filesystem is mounted in NOATIME mode - access times will NEVER be updated automatically")
    elif (not changed_after_simple_access) and changed_after_old_atime:
        print("Your filesystem is mounted in RELATIME mode - access times will only be updated if they are too old")
    elif changed_after_simple_access and (not changed_after_old_atime):
        print("Unable to determine your access time mode")
    else: # Both updated
        print("Your filesystem is mounted in STRICTATIME mode - access times will be updated on EVERY file access")
finally:
    # Delete our test file
    try:
        os.remove("test.txt")
    except:
        pass

Also available on GitHub.

Posted by Uli Köhler in Linux, Python, Windows

How to fix Python “TypeError: utime() takes no keyword arguments”

Problem:

You are trying to set file access/modification dates using os.utime() like this:

os.utime(filename, times=(myatime, mymtime))

but you see an error message like this:

Traceback (most recent call last):
  File "utime.py", line 6, in <module>
    os.utime("myfile.txt", times=(myatime, mymtime))
TypeError: utime() takes no keyword arguments

Solution:

Using os.utime(..., times=(...)) is Python 3 syntax, so use Python 3 if you can!

In case you still need to use Python 2.x, just remove times= from the source code:

os.utime(filename, (myatime, mymtime))

This code will work fine with both Python 2 and Python 3.

Note that Python 2 retirement is just a few months away at the time of writing this post!

Posted by Uli Köhler in Python

How to convert datetime to time float (unix timestamp) in Python 2

Use this snippet to convert a datetime.datetime object into a float (like the one time.time() returns) in Python 2.x:

from datetime import datetime
import time

dt = datetime.now()
timestamp = time.mktime(dt.timetuple()) + dt.microsecond/1e6

After running this, timestamp will be 1563812373.1795 for example.

or use this function:

from datetime import datetime
import time

def datetime_to_timestamp(dt):
    return time.mktime(dt.timetuple()) + dt.microsecond/1e6

In Python 3, you can simply use

dt.timestamp()

but that is not supported in Python 2.

Posted by Uli Köhler in Python

How to fix Python error “AttributeError: ‘datetime.datetime’ object has no attribute ‘timestamp'”

Problem:

You want to convert a datetime object into a unix timestamp (int or float: seconds since 1970-1-1 00:00:00) in Python using code like

from datetime import datetime
timestamp = datetime.now().timestamp()

but you see an error message like this:

Traceback (most recent call last):
  File "unix-timestamp.py", line 2, in <module>
    timestamp = datetime.now().timestamp()
AttributeError: 'datetime.datetime' object has no attribute 'timestamp'

Solution:

You are running your code with Python 2.x which does not support datetime.timestamp() – in most cases the easiest way to fix this issue is to use Python 3, e.g.:

python3 unix-timestamp.py

In case that is not possible e.g. due to incompatibilities, use this snippet instead, which is compatible with both Python 2 and Python 3:

from datetime import datetime
import time

dt = datetime.now()
timestamp = time.mktime(dt.timetuple()) + dt.microsecond/1e6

 

Posted by Uli Köhler in Python

How to fix MongoDB ‘exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /var/lib/mongodb, terminating’

Problem:

When trying to start MongoDB e.g. using sudo systemctl start mongod, the process terminates with status code 100 and you see this error message in /var/log/mongodb/mongodb.log:

2019-07-22T17:11:07.858+0200 I STORAGE  [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /var/lib/mongodb, terminating

Solution:

Fix the permissions of /var/lib/mongodb:

sudo chown -R mongodb: /var/lib/mongodb

then restart MongoDB e.g. using

sudo systemctl restart mongod

In case that does not help, check your error message if you are using a data directory different to /var/lib/mongodb. In that case run

sudo chown -R mongodb: <insert your data directory here>

In case that is not the case or doesn’t help either, check if your filesystem is mounted in read-only mode.

Posted by Uli Köhler in MongoDB

CMake executable minimal example

This CMakeLists.txt builds an executable called main from main.cpp:

add_executable(main main.cpp)

 

Posted by Uli Köhler in CMake

boost::lexical_cast minimal example

#include <boost/lexical_cast.hpp>
#include <iostream>

int main() {
    int a = boost::lexical_cast<int>("123");
    int b = boost::lexical_cast<int>("456");
    
    int c = a + b;
    std::cout << c << std::endl; //Prints 579
}

 

Posted by Uli Köhler in Boost, C/C++

How to set file modification time (mtime) in Python

Also see: How to set file access time (atime) in Python

You can use os.utime() to set the access and modification times of files in Python. In order to set just the access time (mtime) use this snippet:

# mtime must be a datetime
stat = os.stat(filename)
# times must have two floats (unix timestamps): (atime, mtime)
os.utime(filename, times=(stat.st_atime, mtime.timestamp()))

Or use this utility function:

from datetime import datetime
import os

def set_file_modification_time(filename, mtime):
    """
    Set the modification time of a given filename to the given mtime.
    mtime must be a datetime object.
    """
    stat = os.stat(filename)
    atime = stat.st_atime
    os.utime(filename, times=(atime, mtime.timestamp()))

Usage example:

# Set the modification time of myfile.txt to 1980-1-1, leave the acess time intact
set_file_modification_time("myfile.txt", datetime(1980, 1, 1, 0, 0, 0))

In case you need to compatible with Python 2.x, use this variant instead:

from datetime import datetime
import os
import time

def datetime_to_timestamp(dt):
    return time.mktime(dt.timetuple()) + dt.microsecond/1e6

def set_file_modification_time(filename, mtime):
    """
    Set the modification time of a given filename to the given mtime.
    mtime must be a datetime object.
    """
    stat = os.stat(filename)
    atime = stat.st_atime
    os.utime(filename, (atime, datetime_to_timestamp(mtime)))

See How to convert datetime to time float (unix timestamp) in Python 2 and How to fix Python error AttributeError: datetime.datetime object has no attribute timestamp for more details on this alternate approach.

In case you have any option of using Python 3.x, I recommend using the Python 3 version listed above, since it’s much more readable, involves less code and (at the time of writing this code), Python 2 will be retired in only a couple of months. I recommend upgrading your scripts with Python 3 compatibility as soon as possible as many other projects have already done.

Posted by Uli Köhler in Python

How to set file access time (atime) in Python

Also see: How to set file modification time (mtime) in Python

You can use os.utime() to set the access and modification times of files in Python. In order to set just the access time (atime) use this snippet:

# atime must be a datetime
stat = os.stat(filename)
# times must have two floats (unix timestamps): (atime, mtime)
os.utime(filename, times=(atime.timestamp(), stat.st_mtime))

Or use this utility function:

from datetime import datetime
import os

def set_file_access_time(filename, atime):
    """
    Set the access time of a given filename to the given atime.
    atime must be a datetime object.
    """
    stat = os.stat(filename)
    mtime = stat.st_mtime
    os.utime(filename, times=(atime.timestamp(), mtime))

Usage example:

# Set the access time of myfile.txt to 1970-1-1, leave the modification time intact
set_file_access_time("myfile.txt", datetime(1970, 1, 1, 0, 0, 0))

In case you need to compatible with Python 2.x, use this variant instead:

from datetime import datetime
import os
import time

def datetime_to_timestamp(dt):
    return time.mktime(dt.timetuple()) + dt.microsecond/1e6

def set_file_access_time(filename, atime):
    """
    Set the access time of a given filename to the given atime.
    atime must be a datetime object.
    """
    stat = os.stat(filename)
    mtime = stat.st_mtime
    os.utime(filename, (datetime_to_timestamp(atime), mtime))

See How to convert datetime to time float (unix timestamp) in Python 2 and How to fix Python error AttributeError: datetime.datetime object has no attribute timestamp for more details on this alternate approach.

In case you have any option of using Python 3.x, I recommend using the Python 3 version listed above, since it’s much more readable, involves less code and (at the time of writing this code), Python 2 will be retired in only a couple of months. I recommend upgrading your scripts with Python 3 compatibility as soon as possible as many other projects have already done.

Posted by Uli Köhler in Python
This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Cookie settingsACCEPTPrivacy &amp; Cookies Policy