Programming languages

Apr

2021

How to fix GCC lots of undefined reference to std:: functions

Problem:

When trying to compile your C++ application, you see lots of undefined reference to messages like

AutoBenchmark.cpp:(.text+0x6a): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x81): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x8a): undefined reference to `std::chrono::_V2::system_clock::now()'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0xcd): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0xe6): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()'
/usr/bin/ld: /tmp/cc356AdP.o: in function `AutoBenchmark::~AutoBenchmark()':
AutoBenchmark.cpp:(.text+0x17e): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x18e): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()'
/usr/bin/ld: /tmp/cc356AdP.o: in function `AutoBenchmark::Record(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
AutoBenchmark.cpp:(.text+0x1db): undefined reference to `std::chrono::_V2::system_clock::now()'
/usr/bin/ld: /tmp/cc356AdP.o: in function `AutoBenchmark::Record(char const*)':
AutoBenchmark.cpp:(.text+0x252): undefined reference to `std::allocator<char>::allocator()'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x288): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x294): undefined reference to `std::allocator<char>::~allocator()'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x2b9): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x2ce): undefined reference to `std::allocator<char>::~allocator()'
/usr/bin/ld: /tmp/cc356AdP.o: in function `AutoBenchmark::Print()':
AutoBenchmark.cpp:(.text+0x316): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::length() const'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x332): undefined reference to `std::cout'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x337): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <char, std::char_traits<char>, std::allocator<char> >(std::basic_ostream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x344): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x572): undefined reference to `std::cout'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x577): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <char, std::char_traits<char>, std::allocator<char> >(std::basic_ostream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x59c): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <char, std::char_traits<char>, std::allocator<char> >(std::basic_ostream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x5ab): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x5c2): undefined reference to `std::ostream::operator<<(double)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x5d1): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x607): undefined reference to `std::cout'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x60c): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <char, std::char_traits<char>, std::allocator<char> >(std::basic_ostream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x631): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <char, std::char_traits<char>, std::allocator<char> >(std::basic_ostream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x640): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x657): undefined reference to `std::ostream::operator<<(double)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x666): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x69c): undefined reference to `std::cout'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x6a1): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <char, std::char_traits<char>, std::allocator<char> >(std::basic_ostream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x6c6): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <char, std::char_traits<char>, std::allocator<char> >(std::basic_ostream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x6d5): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x6ec): undefined reference to `std::ostream::operator<<(double)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x6fb): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x731): undefined reference to `std::cout'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x736): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <char, std::char_traits<char>, std::allocator<char> >(std::basic_ostream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x75b): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <char, std::char_traits<char>, std::allocator<char> >(std::basic_ostream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x76a): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x781): undefined reference to `std::ostream::operator<<(double)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x790): undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)'
/usr/bin/ld: AutoBenchmark.cpp:(.text+0x7c6): undefined reference to `std::cout'

Solution:

You need to link the stdc++ library by adding -lstdc++ to your compiler/linker flags.

For example, instead of

gcc -o myprogram *.c

use

gcc -o myprogram *.c -lstdc++

Apr

2021

How to fix GCC error: expected constructor, destructor, or type conversion before __declspec(dllexport)

Problem:

When trying to compile an application on Linux, you see a compiler error message like

myheader.h:23:12: error: expected constructor, destructor, or type conversion before ‘(’ token
   23 |  __declspec(dllexport) int myfunc(

Solution:

__declspec(dllexport) is a Windows-specific feature and not available on Linux. In order to fix it in a compatible way, add this code either in a header that is included in every file containing __declspec(dllexport) or add it in each file where the error occurs:

#ifdef __linux__
#define __declspec(v)
#endif

This will basically ignore any __declspec() call on the preprocessor level.

Apr

2021

How to fix GCC undefined reference to `sqrt’

Problem:

When trying to compile your application using gcc, you see an error message like

/usr/bin/ld: /tmp/ccxPIowU.o: in function `run_mymath':
mathstuff.c:(.text+0x15d): undefined reference to `sqrt'

Solution:

You need to link the math library using the -lm flag (-lxxx means: “link the xxx library”, i.e. -lm means “link the m” library)

For example, instead of

gcc -o myprogram *.c

use

gcc -o myprogram *.c -lm

Apr

2021

How to fix GCC undefined reference to `_finite’ or implicit declaration of function ‘_finite’

Problem:

If you’re trying to compile Windows code on Linux, you will often see messages like

lmmin.c:261:10: warning: implicit declaration of function ‘_finite’; did you mean ‘finite’? [-Wimplicit-function-declaration]
  116 |     if(!_finite(myvalue)){

/usr/bin/ld: mymath.c:(.text+0x1057): undefined reference to `_finite'

and your code won’t compile.

Solution:

_finite is a function that is only available on Windows. In order to use it on Linux using GCC or G++, one option is to use isfinite() from math.h:

#include <math.h>

#define _finite(v) (isfinite((v)))

In case that function is not available (like on some Microcontroller platforms), you can use __builtin_isfinite(). Note that glibc defines isfinite() as an alias for __builtin_isfinite(). Use it like this:

#define _finite(v) (__builtin_isfinite(v))

In case you want to have code that is compatible with both platforms (Windows and Linux), use

#ifdef __linux__
#define _finite(v) (__builtin_isfinite(v))
#endif

#include <math.h>

#ifdef __linux__
#define _finite(v) (isfinite((v)))
#endif

Apr

2021

How to ignore pylint problem for a specific single line of code

If you have a Python line of code like

writer = pd.ExcelWriter(filename)

that produces a pylint problem message like

Abstract class 'ExcelWriter' with abstract methods instantiated pylint(abstract-class-instantiated)

you can ignore it by adding a comment in the format # pylint: disable=[problem-code] at the end of the line where [problem-code] is the value inside pylint(...) in the pylint message – for example, abstract-class-instantiated for the problem report listed above.

The modified line will look like this:

writer = pd.ExcelWriter(filename) # pylint: disable=abstract-class-instantiated

You can also add the comment line

# pylint: disable=abstract-class-instantiated

to the function-level (or to any other block-level) to disable that pylint rule for the entire current function or block.

Apr

2021

How to migrate from ansicolor to ansicolors

The ansicolor package in Python has not been updated in a long time and doesn’t have any documentation on PyPI. Therfore, I migrated my Python scripts to ansicolors (with an s at the end). Only a few steps are neccessary:

First, replace

from ansicolor import red

from colors import red

Additionally, bold=True is called style="bold" in ansicolors, hence you need to replace

red("msg", bold=True)

red("msg", style="bold")

Apr

2021

How to fix matplotlib .ylabel() AttributeError: ‘AxesSubplot’ object has no attribute ‘ylabel’

Problem:

You want to set the ylabel of a matplotlib plot using .ylabel("My ylabel") but you see an error messsage like

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-12-28a87d95bccc> in <module>
---> 11 axs[0].ylabel("My ylabel")

AttributeError: 'AxesSubplot' object has no attribute 'ylabel'

Solution:

Use .set_ylabel("My ylabel") instead of .ylabel("My ylabel") !

While

from matplotlib import pyplot as plt

plt.ylabel("My ylabel")

works fine, if you have an axes object like the one you get from plt.subplots(), you’ll have to use set_ylabel()!

from matplotlib import pyplot as plt
fig, axs = plt.subplots(2, 1)
# ...
axs[0].set_ylabel("My ylabel")

Apr

2021

How to fix matplotlib .xlabel() AttributeError: ‘AxesSubplot’ object has no attribute ‘xlabel’

Problem:

You want to set the xlabel of a matplotlib plot using .xlabel("My xlabel") but you see an error messsage like

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-12-28a87d95bccc> in <module>
---> 11 axs[0].xlabel("My xlabel")

AttributeError: 'AxesSubplot' object has no attribute 'xlabel'

Solution:

Use .set_xlabel("My xlabel") instead of .xlabel("My xlabel") !

While

from matplotlib import pyplot as plt

plt.xlabel("My xlabel")

works fine, if you have an axes object like the one you get from plt.subplots(), you’ll have to use set_xlabel()!

from matplotlib import pyplot as plt
fig, axs = plt.subplots(2, 1)
# ...
axs[0].set_xlabel("My xlabel")

Apr

2021

How to fix matplotlib .title() TypeError: ‘Text’ object is not callable

Problem:

You want to set the title of a matplotlib plot using .title("My title") but you see an error messsage like

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-11-f5f930f00eac> in <module>
---> 10 axs[0].title("My title")

TypeError: 'Text' object is not callable

Solution:

Use .set_title("My title") instead of .title("My title") !

While

from matplotlib import pyplot as plt

plt.title("My title")

works fine, if you have an axes object like the one you get from plt.subplots(), you’ll have to use set_title()!

from matplotlib import pyplot as plt
fig, axs = plt.subplots(2, 1)
# ...
axs[0].set_title("My title")

Apr

2021

How to replace string in column names in Pandas DataFrame

Use this snippet in order to replace a string in column names for a pandas DataFrame:

new_df = df.rename(columns=lambda s: s.replace("A", "B")) # df will not be modified !

You can also modify the column names in-place (i.e. modify the original DataFrame):

df.rename(columns=lambda s: s.replace("A", "B"), inplace=True)

For example, if you have the columns ["ColumnA", "X", "Y"] before running .rename(), the result will have ["ColumnB", "X", "Y"] (the "A" has been replaced by "B")

Apr

2021

How to find out if BytesIO is empty in Python

In order to find out if a BytesIO instance is empty or not, get its size and then check if it’s > 0:

my_bytesio.getbuffer().nbytes > 0

The following example shows how to use it in an if clause:

if my_bytesio.getbuffer().nbytes > 0:
    print("my_bytesio is empty")
else:
    print("my_bytesio is NOT empty")

Apr

2021

How to get size of BytesIO in Python

If you want to find out the size of the data stored in a io.BytesIO instance in Python, use

my_bytesio.getbuffer().nbytes

Mar

2021

How to send email with BytesIO attachment via SMTP in Python

This example details how to send an email in Python, with an attachment from a io.BytesIO instance instead of reading the attachment from a file on the filesystem:

#!/usr/bin/env python3
__author__ = "Uli Köhler"
__license__ = "CC0 1.0 Universal (public domain)"
__version__ = "1.0"
import smtplib
import mimetypes
from io import BytesIO
from email.message import EmailMessage

# Create message and set text content
msg = EmailMessage()
msg['Subject'] = 'This email contains an attachment'
msg['From'] = '[email protected]'
msg['To'] = '[email protected]'
# Set text content
msg.set_content('Please see attached file')

def attach_bytesio_to_email(email, buf, filename):
    """Attach a file identified by filename, to an email message"""
    # Reset read position & extract data
    buf.seek(0)
    binary_data = buf.read()
    # Guess MIME type or use 'application/octet-stream'
    maintype, _, subtype = (mimetypes.guess_type(filename)[0] or 'application/octet-stream').partition("/")
    # Add as attachment
    email.add_attachment(binary_data, maintype=maintype, subtype=subtype, filename=filename)

# Attach files
buf = BytesIO()
buf.write(b"This is a test text")
attach_bytesio_to_email(msg, buf, "test.txt")

def send_mail_smtp(mail, host, username, password):
    s = smtplib.SMTP(host)
    s.starttls()
    s.login(username, password)
    s.send_message(msg)
    s.quit()

send_mail_smtp(msg, 'smtp.my-domain.com', '[email protected]', 'sae7ooka0S')

The script from above is using the following utility functions:

def attach_bytesio_to_email(email, buf, filename):
    """Attach a file identified by filename, to an email message"""
    # Reset read position &amp; extract data
    buf.seek(0)
    binary_data = buf.read()
    # Guess MIME type or use 'application/octet-stream'
    maintype, _, subtype = (mimetypes.guess_type(filename)[0] or 'application/octet-stream').partition("/")
    # Add as attachment
    email.add_attachment(binary_data, maintype=maintype, subtype=subtype, filename=filename

def send_mail_smtp(mail, host, username, password):
    s = smtplib.SMTP(host)
    s.starttls()
    s.login(username, password)
    s.send_message(msg)
    s.quit()

which you can use in your code directly. The easiest way to initialize the email message is using

# Create message and set text content
msg = EmailMessage()
msg['Subject'] = 'This email contains an attachment'
msg['From'] = '[email protected]'
msg['To'] = '[email protected]'
# Set text content
msg.set_content('Please see attached file')

and then attaching your BytesIO instance (named buf) using

attach_bytesio_to_email(msg, buf, "test.txt")

and once you’re finished with adding attachments, sending the message using

send_mail_smtp(msg, 'smtp.my-domain.com', '[email protected]', 'sae7ooka0S')

Mar

2021

How to send email with file attachment via SMTP in Python

This example shows how to send an email with an attachment in Python, with the attachment being read from a file from the filesystem:

#!/usr/bin/env python3
import smtplib
import mimetypes
from email.message import EmailMessage

# Create message and set text content
msg = EmailMessage()
msg['Subject'] = 'This email contains an attachment'
msg['From'] = '[email protected]'
msg['To'] = '[email protected]'
# Set text content
msg.set_content('Please see attached file')

def attach_file_to_email(email, filename):
    """Attach a file identified by filename, to an email message"""
    with open(filename, 'rb') as fp:
        file_data = fp.read()
        maintype, _, subtype = (mimetypes.guess_type(filename)[0] or 'application/octet-stream').partition("/")
        email.add_attachment(file_data, maintype=maintype, subtype=subtype, filename=filename)

# Attach files
attach_file_to_email(msg, "myfile.pdf")

def send_mail_smtp(mail, host, username, password):
    s = smtplib.SMTP(host)
    s.starttls()
    s.login(username, password)
    s.send_message(mail)
    s.quit()

send_mail_smtp(msg, 'smtp.my-domain.com', '[email protected]', 'sae7ooka0S')

The utility functions in this code are:

import smtplib
import mimetypes

def attach_file_to_email(email, filename):
    """Attach a file identified by filename, to an email message"""
    with open(filename, 'rb') as fp:
        file_data = fp.read()
        maintype, _, subtype = (mimetypes.guess_type(filename)[0] or 'application/octet-stream').partition("/")
        email.add_attachment(file_data, maintype=maintype, subtype=subtype, filename=filename)

def send_mail_smtp(mail, host, username, password):
    s = smtplib.SMTP(host)
    s.starttls()
    s.login(username, password)
    s.send_message(mail)
    s.quit()

Initialize your email like this:

# Create message and set text content
msg = EmailMessage()
msg['Subject'] = 'This email contains an attachment'
msg['From'] = '[email protected]'
msg['To'] = '[email protected]'
# Set text content
msg.set_content('Please see attached file')

and then attach the file like this:

attach_file_to_email(msg, "myfile.pdf")

and send the email using

send_mail_smtp(msg, 'smtp.my-domain.com', '[email protected]', 'sae7ooka0S')

Mar

2021

How to download Wasabi/S3 object to string/bytes using boto3 in Python

You can use io.BytesIO to store the content of an S3 object in memory and then convert it to bytes which you can then decode to a str. The following example downloads myfile.txt into memory:

# Download to file
buf = io.BytesIO()
my_bucket.download_fileobj("myfile.txt", buf)
# Get file content as bytes
filecontent_bytes = buf.getvalue()
# ... or convert to string
filecontent_str = buf.getvalue().decode("utf-8")

Full example

import boto3
import io

# Create connection to Wasabi / S3
s3 = boto3.resource('s3',
    endpoint_url = 'https://s3.eu-central-1.wasabisys.com',
    aws_access_key_id = 'MY_ACCESS_KEY',
    aws_secret_access_key = 'MY_SECRET_KEY'
)

# Get bucket object
my_bucket = s3.Bucket('boto-test')

# Download to file
buf = io.BytesIO()
my_bucket.download_fileobj("myfile.txt", buf)
# Get file content as bytes
filecontent_bytes = buf.getvalue()
# ... or convert to string
filecontent_str = buf.getvalue().decode("utf-8")

print(filecontent_str)

Don’t forget to fill in MY_ACCESS_KEY and MY_SECRET_KEY. Depending on what region and what S3-compatible service you use, you might need to use another endpoint URL instead of https://s3.eu-central-1.wasabisys.com.

Mar

2021

How to upload string as Wasabi/S3 object using boto3 in Python

In order to upload a Python string like

my_string = "This shall be the content for a file I want to create on an S3-compatible storage"

to an S3-compatible storage like Wasabi or Amazon S3, you need to encode it using .encode("utf-8") and then wrap it in an io.BytesIO object:

my_bucket.upload_fileobj(io.BytesIO(my_string.encode("utf-8")), "myfile.txt")

Full example:

import boto3
import io

# Create connection to Wasabi / S3
s3 = boto3.resource('s3',
    endpoint_url = 'https://s3.eu-central-1.wasabisys.com',
    aws_access_key_id = 'MY_ACCESS_KEY',
    aws_secret_access_key = 'MY_SECRET_KEY'
)

# Get bucket object
my_bucket = s3.Bucket('boto-test')

# Upload string to file
my_string = "This shall be the content for a file I want to create on an S3-compatible storage"

my_bucket.upload_fileobj(io.BytesIO(my_string.encode("utf-8")), "myfile.txt")

Mar

2021

Mini CSS cheat-sheet for editing websites

These are the most common CSS attributes I use for editing existing websites

General

Append !important if your style doesn’t get applied. This overrides other styles, e.g. display: none !important;

display: none, block, inline-block

Colors & font style

Text color

color: red, #FF0000, lightblue – goto https://colorpicker.me/ and then copy the hex code

Color of the block behind the text:

background-color: red, #FF0000, lightblue – goto https://colorpicker.me/ and then copy the hex code

Other font styling:

font-weight: normal, bold

font-size: 200%, 30px

Spacing

Space outside the element (to other elements)

margin-top / margin-left / margin-right / margin-bottom: 20px

Space inside the element, to its content:

padding-top / padding-left / padding-right / padding-bottom: 20px

Geometric properties

Make the corners of a block rounded instead of sharp:

border-radius: 5px

Width and height of the element

width: 100%, 300px, 30vw

height: 30px, 100%

Mar

2021

How to filter for objects in a given S3 directory using boto3

Using boto3, you can filter for objects in a given bucket by directory by applying a prefix filter.

Instead of iterating all objects using

for obj in my_bucket.objects.all():
    pass # ...

(see How to use boto3 to iterate ALL objects in a Wasabi / S3 bucket in Python for a full example)

you can apply a prefix filter using

for obj in my_bucket.objects.filter(Prefix="MyDirectory/"):
    print(obj)

Don’t forget the trailing / for the prefix argument ! Just using filter(Prefix="MyDirectory") without a trailing slash will also match e.g. MyDirectoryFileList.txt.

This complete example prints the object description for every object in the 10k-Test-Objects directory (from our post on How to use boto3 to create a lot of test files in Wasabi / S3 in Python).

import boto3

# Create connection to Wasabi / S3
s3 = boto3.resource('s3',
    endpoint_url = 'https://s3.eu-central-1.wasabisys.com',
    aws_access_key_id = 'MY_ACCESS_KEY',
    aws_secret_access_key = 'MY_SECRET_KEY'
)

# Get bucket object
my_bucket = s3.Bucket('boto-test')

# Iterate over objects in bucket
for obj in my_bucket.objects.filter(Prefix="MyDirectory"):
    print(obj)

Example output:

s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/1.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/10.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/100.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/1000.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/10000.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/1001.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/1002.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/1003.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/1004.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/1005.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/1006.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/1007.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/1008.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/1009.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/101.txt')
[...]

Mar

2021

How to use boto3 to iterate ALL objects in a Wasabi / S3 bucket in Python

This snippet shows you how to iterate over all objects in a bucket:

import boto3

# Create connection to Wasabi / S3
s3 = boto3.resource('s3',
    endpoint_url = 'https://s3.eu-central-1.wasabisys.com',
    aws_access_key_id = 'MY_ACCESS_KEY',
    aws_secret_access_key = 'MY_SECRET_KEY'
)

# Get bucket object
my_bucket = s3.Bucket('boto-test')

# Iterate over objects in bucket
for obj in my_bucket.objects.all():
    print(obj)

Example output:

s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/1.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/10.txt')
s3.ObjectSummary(bucket_name='boto-test', key='10k-Test-Objects/100.txt')
[...]

Mar

2021

How to use boto3 to create a lot of test files in Wasabi / S3 in Python

The following example code creates 10000 test files on Wasabi / S3. It is based on How to use concurrent.futures map with a tqdm progress bar:

import boto3
import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(64)

from tqdm import tqdm
import concurrent.futures
def tqdm_parallel_map(executor, fn, *iterables, **kwargs):
    """
    Equivalent to executor.map(fn, *iterables),
    but displays a tqdm-based progress bar.
    
    Does not support timeout or chunksize as executor.submit is used internally
    
    **kwargs is passed to tqdm.
    """
    futures_list = []
    for iterable in iterables:
        futures_list += [executor.submit(fn, i) for i in iterable]
    for f in tqdm(concurrent.futures.as_completed(futures_list), total=len(futures_list), **kwargs):
        yield f.result()

# Create connection to Wasabi / S3
s3 = boto3.resource('s3',
    endpoint_url = 'https://s3.eu-central-1.wasabisys.com',
    aws_access_key_id = 'MY_ACCESS_KEY',
    aws_secret_access_key = 'MY_SECRET_KEY'
)

# Get bucket object
boto_test_bucket = s3.Bucket('boto-test')

def create_s3_object(i, directory):
    # Create test data
    buf = io.BytesIO()
    buf.write(f"{i}".encode())
    # Reset read pointer. DOT NOT FORGET THIS, else all uploaded files will be empty!
    buf.seek(0)

    # Upload the file
    boto_test_bucket.upload_fileobj(buf, f"{directory}/{i}.txt")

for _ in tqdm_parallel_map(executor, lambda i: create_s3_object(i, directory="10k-Test-Objects"), range(1, 10001)):
    pass

Don’t forget to fill in MY_ACCESS_KEY and MY_SECRET_KEY. Depending on what region and what S3-compatible service you use, you might need to use another endpoint URL at https://s3.eu-central-1.wasabisys.com.

Note that running this script, especially when creating lots of test files, will send a lot of requests to your S3 provider and, depending on what plan you are using, these requests might be expensive. Wasabi, for example, does not charge for requests but charges for storage (with a minimum of 1TB storage per month being charged, at the time of writing this).

Problem:

Solution:

Problem:

Solution:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Full example

Full example:

General

Colors & font style

Spacing

Geometric properties

Search

Categories