Converting namedtuples to XLSX in Python

This Python snippet allows you to convert an iterable of namedtuple instances to an XLSX file using xlsxwriter.

The header is automatically determined from the first element of the iterable. If the iterable is empty, the resulting XLSX file will also be empty.

import xlsxwriter
import itertools
from collections import namedtuple

def xlsx_write_rows(filename, rows):
"""
Write XLSX rows from an iterable of rows.
Each row must be an iterable of writeable values.

Returns the number of rows written
"""
workbook = xlsxwriter.Workbook(filename)
# Write values
nrows = 0
for i, row in enumerate(rows):
for j, val in enumerate(row):
worksheet.write(i, j, val)
nrows += 1
# Cleanup
workbook.close()
return nrows

def namedtuples_to_xlsx(filename, values):
"""
Convert a list or generator of namedtuples to an XLSX file.
Returns the number of rows written.
"""
try:
# Ensure its a generator (next() not allowed on lists)
values = (v for v in values)
# Use first row to generate header
peek = next(values)
except StopIteration:  # Empty generator
# Write empty xlsx
return xlsx_write_rows(filename, [])


Example Usage:

MyType = namedtuple("MyType", ["ID", "Name", "Value"])
namedtuples_to_xlsx("test.xlsx", [
MyType(1, "a", "b"),
MyType(2, "c", "d"),
MyType(3, "e", "f"),
])

This example will generate this table:

ID	Name	Value
1	a	b
2	c	d
3	e	f

Iterating prime numbers in Python

Problem:

Using Python you want to iterate through consecutive prime numbers

Solution:

You can use gmpy2‘s next_prime() to do this:

import gmpy2

def primes(start=2):
n = start
while True:
n = gmpy2.next_prime(n)
yield n

# Usage example 1
for prime in primes():
print(prime)

# Usage example 2
for i, prime in enumerate(primes()):
print("The {}th prime number is {}".format(i, prime))

Fixing TensorFlow libcublas.so.8.0: cannot open shared object file on Ubuntu

Problem:

When you run import tensorflow in Python, you get one of the following errors:

ImportError: libcublas.so.8.0: cannot open shared object file: No such file or directory
ImportError: libcusolver.so.8.0: cannot open shared object file: No such file or directory
ImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory
ImportError: libcufft.so.8.0: cannot open shared object file: No such file or directory
ImportError: libcurand.so.8.0: cannot open shared object file: No such file or directory

Solution:

Install the required packages using:

apt-get install libcublas8.0 libcusolver8.0 libcudart8.0 libcufft8.0 libcurand8.0

Note that you also need to install cuDNN – see this followup post

Which version on CuDNN should you install for TensorFlow GPU on Ubuntu?

for details on how to do that.

If this method does not work, you can (as a quick workaround) uninstall tensorflow-gpu and install the tensorflow – the version without GPU support:

pip3 uninstall tensorflow-gpu
pip3 install tensorflow

However, this will likely make your applications much slower.

For other solutions see the TensorFlow bugtracker on GitHub.

How to use concurrent.futures map with a tqdm progress bar

Problem:

You have a concurrent.futures executor, e.g.

import concurrent.futures

executor = concurrent.futures.ThreadPoolExecutor(64)

Using this executor, you want to map a function over an iterable in parallel (e.g. parallel download of HTTP pages).

In order to aid interactive execution, you want to use tqdm to provide a progress bar, showing the fraction of futures

Solution:

You can use this function:

from tqdm import tqdm
import concurrent.futures

def tqdm_parallel_map(executor, fn, *iterables, **kwargs):
"""
Equivalent to executor.map(fn, *iterables),
but displays a tqdm-based progress bar.

Does not support timeout or chunksize as executor.submit is used internally

**kwargs is passed to tqdm.
"""
futures_list = []
for iterable in iterables:
futures_list += [executor.submit(fn, i) for i in iterable]
for f in tqdm(concurrent.futures.as_completed(futures_list), total=len(futures_list), **kwargs):
yield f.result()


Note that internally, executor.submit() is used, not executor.map() because there is no way of calling concurrent.futures.as_completed() on the iterator returned by executor.map().

Note: In constract to executor.map() this function does NOT yield the arguments in the same order as the input.

Problem:

You want to download a URL to a file using the requests python library, but you want to skip the download if it doesn’t exist

Solution:

Use the following functions:

import requests
import os.path

"""
"""
with open(filename, 'wb') as fout:
response = requests.get(url, stream=True)
response.raise_for_status()
# Write response data to file
for block in response.iter_content(4096):
fout.write(block)

"""

Returns
-------
"""
if not os.path.exists(filename):
return True
return False


Computing bounding box for a list of coordinates in Python

Problem:

You have a list of X/Y coordinates, for example:

coords = [(6.74219, -53.57835),
(6.74952, -53.57241),
(6.75652, -53.56289),
(6.74756, -53.56598),
(6.73462, -53.57518)]

For these coordinates you want to compute the minimum bounding box.

Removing spans/divs with style attributes from HTML

Occasionally I have to clean up some HTML code – mostly because parts of it were pasted into a CMS like WordPress from rich text editor like Word.

I’ve noticed that the formatting I want to remove is mostly based on span and div elements with a style attribute. Therefore, I’ve written a simple Python script based on BeautifulSoup4 which will replace certain tags with their contents if they have a style attribute. While in some cases some other formatting might be destroyed by such a script, it is very useful for some recurring usecases.

Computing the LP2980 adjust resistor using Python

The LP2980ADJ is a 50 mA LDO that be configured for an output voltage from 1.23V to 15V using a pair of resistors.

The datasheet lists a formula for the output voltage, however no easy-to-use customizable software is provided that can be used to directly compute the correct resistor in a reproducible way. Read more

The following html code can be used to create an html form that allows uploading multiple files at once:

<form enctype="multipart/form-data" method="POST" action="upload.py">
<table style="width: 100%">
<tr>
<td style="text-align: right"><input type="file" multiple="" id="files" name="files"></td>
</tr>
<tr>
<td></td>
</tr>
</table>
</form>

Normalizing electronics engineering value notations using Python

In electronics engineering there is a wide variety of notations for values that need to be recognized by intuitive user interfaces. Examples include:

• 1fA
• 0.1A
• 0.00001
• 1e-6
• 4,5nA
• 4,500.123 A
• 4A5
• 4k0 A

The wide variety of options, including thousands separators, comma-as-decimal-separator and suffix-as-decimal-separator, optional whitespace and scientific notations makes it difficult to normalize values without using specialized libraries. Read more

Calculating the NCP380 Ilim resistor using Python

Problem

You want to calculate the correct value for the Ilim resistor for the NCP380 current limiter IC with a custom current limit. Read more

Finding the nearest E96 resistor value in Python

Problem

You want to find the E24/E48/E96 resistor value that is closest to a given exact value programmatically using python. Read more

Engineering for the super-lazy: Solving equations without activating your brain

Preface

In electronics engineering, from time to time you have to use standard formulas to characterize your circuits. To what extent you need to calculate all parameters most often depends on the requirement.

For example, consider the formula for the -3dB cutoff frequency of a 1st order RC lowpass filter:

$$f_c=\frac{1}{2\pi RC}$$

Although this equation is fairly simple and most people won’t have any problem solving it for any particular variable in a few seconds, it can serve as a basic example on how to solve an equation symbolically.

One of the easiest ways of performing this task is to use SymPy, a Python library for symbolic mathematics.

Parsing the MeSH ASCII format in Python

Similar to our previously published UniPruot parser, MeSH provides an ASCII format that can easily be parsed using Python.

Just like the UniProt parser, this function yields MeSH entries represented by dictionaries. Example code is included at the bottom of the file.

Fixing ImportError: cannot import name ‘urlencode’ in Python3

Problem:

Your python 3.x interpreter prints the error message

ImportError: cannot import name 'urlencode'

Parsing OpenGeoDB PLZ code TSV files using Python

Problem:

You want to parse the OpenGeoDB Postleitzahl (german ZIP code) TSV files available from here.

Using python requests over Tor

Problem:

You want to use the python requests library over a Tor connection using the Tor builtin SOCKS proxy.

Fixing bad blocks on HDDs using fixhdd.py

Problem:

You hard drive or SMART tool reports errors when reading specific blocks similar to this message:

[3142.686141] end_request: I/O error, dev sda, sector 31415926

No matter how often you read the block, the hard drive still returns an error and does not reallocate the block.

Listing files inside a Torrent

Torrent files are essentially containers that store information about a set of files to be downloaded via P2P.

Unfortunately it is not easily possible to simply list the files that are stored in a torrent. Using our python script torrentls.py you can easily accomplish that task.

Reading the UniProt text format in Python

Just like many other databases in computational biology, the downloads for the popular UniProt database are available in a custom text format which is documented on ExPASy.

While it is certainly not difficult writing a generic parser and one can use BioPython, I believe it is often easier to use a tested parser that only uses standard libraries.