Reading and writing KiCAD libraries in Python

This snippet provides a parser and serializer for KiCAD schematic symbol libraries and DCM symbol documentation libraries. It will parse the KiCAD libraries into a number of records, consisting of a list of lines. An extended version of this snippet has been used in my KiCADSamacSysImporter project

__author__ = "Uli Köhler"
__license__ = "CC0 1.0 Universal"

class KiCADDocLibrary(object):
    def __init__(self, records=[]):
        self.records = records
        
    @staticmethod
    def read(file):
        # A record is everything between '$CMP ...' line and '$ENDCMP' line
        records = []
        current_record = None
        for line in file:
            line = line.strip()
            if line.startswith('$CMP '):
                current_record = []
            # Add line to record if we have any current record
            if current_record is not None:
                current_record.append(line)
            if line.startswith("$ENDCMP"):
                if current_record is not None:
                    records.append(current_record)
                    current_record = None
        return KiCADDocLibrary(records)
        
    def write(self, out):
        # Write header
        out.write("EESchema-DOCLIB  Version 2.0\n")
        # Write records
        for rec in self.records:
            out.write("#\n")
            for line in rec:
                out.write(line)
                out.write("\n")
        # Write footer
        out.write("#\n#End Doc Library\n")        

        
class KiCADSchematicSymbolLibrary(object):
    def __init__(self, records=[]):
        self.records = records
        
    @staticmethod
    def read(file):
        # A record is everything between 'DEF ...' line and 'ENDDEF' line
        records = []
        current_record = None
        current_comment_lines = [] # 
        for line in file:
            line = line.strip()
            if line.startswith("#encoding"):
                continue # Ignore - we always use #encoding utf-8
            # Comment line processing
            if line.startswith("#"):
                current_comment_lines.append(line)
            # Start of record
            if line.startswith('DEF '):
                current_record = current_comment_lines
                current_comment_lines = []
            # Add line to record if we have any current record
            if current_record is not None:
                current_record.append(line)
            if line.startswith("ENDDEF"):
                if current_record is not None:
                    records.append(current_record)
                    current_record = None
            # Clear comment lines
            # We can only do this now to avoid clearing them before
            #  they are used in the DEF clause
            if not line.startswith("#"):
                current_comment_lines = []
        return KiCADSchematicSymbolLibrary(records)
        
    def write(self, out):
        # Write header
        out.write("EESchema-LIBRARY Version 2.4\n#encoding utf-8\n")
        # Write records
        for rec in self.records:
            for line in rec:
                out.write(line)
                out.write("\n")
        # Write footer
        out.write("#\n#End Library\n")        

 

Posted by Uli Köhler in Electronics, KiCAD, Python

What does ‘+ve’ or ‘-ve’ mean in a medical context?

+ve means positive
-ve means negative

Some publishers use these terms instead of just + and - to make clear that the + indicates that a certain parameter is positive or negative, to avoid confusing - with a dash and + with and.

For example, writing p53-ve clearly indicates that p53 is negative. What positive or negative actually means depends on the context, e.g. it can mean that p53 is not mutated in a group of patients. See this paper for an example where p53-ve is used in that way.

 

Posted by Uli Köhler in Bioinformatics

Python example: List files in ZIP archive

This example lists all the filenames contained in a ZIP file using Python’s built-in zipfile library.

#!/usr/bin/env python3
import zipfile

def list_files_in_zip(filename):
    """
    List all files inside a ZIP archive
    Yields filename strings.
    """
    with zipfile.ZipFile(filename) as thezip:
        for zipinfo in thezip.infolist():
            yield zipinfo.filename
                
# Usage example
for filename in list_files_in_zip("myarchive.zip"):
    print(filename)

 

Posted by Uli Köhler in Python

How to initialize your KiCAD project on the command line

Note: This script initializes a KiCAD project in the recommended configuration (i.e. with project-specific footprint and symbol libraries). In case you want to initialize an empty project, see How to initialize an empty KiCAD project on the command line

TL;DR:

Inside the directory where you want to create the project, run

wget -qO- https://techoverflow.net/scripts/kicad-init.sh | bash /dev/stdin MyProject

You should replace MyProject (at the end of the command) with your project name.

Note: This will initialize an empty KiCAD project without any libraries. This is equivalent to creating a new project in KiCAD itself (using the GUI).

Continue reading →

Posted by Uli Köhler in Electronics, KiCAD, Shell

bash: Pass arguments but read script from stdin

You can use /dev/stdin to read the bash script from stdin but still pass command line arguments:

cat myscript.sh | bash /dev/stdin arg1 arg2 # ...

This is especially useful when directly piping scripts from wget or curl. Example:

wget https://example.com/script.sh | bash /dev/stdin arg

 

 

Posted by Uli Köhler in Shell

How to initialize an empty KiCAD project on the command line

TL;DR:

Note: We recommend to use our new script to initialize a project with project-specific footprint and symbol libraries, see How to initialize your KiCAD project on the command line. The script on this page initializes an empty project without any libraries.

Inside the directory where you want to create the project, run

wget -qO- https://techoverflow.net/scripts/kicad-initialize.sh | bash /dev/stdin MyProject

You should replace MyProject (at the end of the command) with your project name.

Note: This will initialize an empty KiCAD project without any libraries. This is equivalent to creating a new project in KiCAD itself (using the GUI).

Continue reading →

Posted by Uli Köhler in Electronics, KiCAD, Shell

bash template: Script with one argument

Use this template to create a bash / shell script with one argument that prints a usage message if used with the wrong number of arguments:

#!/bin/bash
if [ $# -ne 1 ]
then
    echo "Usage: $0 <filename>"
    exit 1
fi
# Note: The argument is $1
# TODO: Place your code here

 

Posted by Uli Köhler in Shell

How to list MongoDB databases on command line

Use this command to list the MongoDB databases on the command line:

echo 'show dbs' | mongo

Example output:

MongoDB shell version v4.0.13
connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("5c1e505e-9b05-4270-ab20-c537c0760481") }
MongoDB server version: 4.0.13
admin       0.000GB
config      0.000GB
drawing     0.001GB
order       0.000GB
production  0.001GB
standards   0.001GB
user        0.000GB
bye

 

Posted by Uli Köhler in MongoDB

How to fix Puppetteer ‘Running as root without –no-sandbox is not supported’

Problem:

When you try to run your puppetteer application, e.g. under docker, you see this error message:

Solution:

Note: Unless you are running in a Docker or similar container, first consider running the application as non-root-user!

You have to pass the --no-sandbox option to puppeteer.launch():

const browser = await puppeteer.launch({
    headless: true,
    args: ['--no-sandbox']
});

We recommend to use this slightly more complex solution to pass it only if the process is being run as root:

/**
 * Return true if the current process is run by the root user
 * https://techoverflow.net/2019/11/07/how-to-check-if-nodejs-is-run-by-root/
 */
function isCurrentUserRoot() {
   return process.getuid() == 0; // UID 0 is always root
}

const browser = await puppeteer.launch({
    headless: true,
    args: isCurrentUserRoot() ? ['--no-sandbox'] : undefined
});

This ensures Chromium is run in the most secure mode possible with the current user.

Posted by Uli Köhler in Javascript, NodeJS, Puppeteer

How to fix Angular Cannot find module ‘@angular/http’ after ‘ng update’

When updating to Angular 8.x+, developers commonly encouter this type of error:

app/app.module.ts:9:42 - error TS2307: Cannot find module '@angular/http'.

9 import {HttpModule, RequestOptions} from '@angular/http';

The solution

The @angular/http module has been removed in recent versions of Angular and replaced by @angular/common/http.  Furthermore, there are some name changes that might need manual fixing, including:

  • HttpModuleHttpClientModule
  • HttpHttpClient
  • URLSearchParams ⇒ HttpParams

Note that the replacements are not always exact equivalent and hence might require some work to get working

See the official @angular/http deprecation guide for more details on how to migrate.

Posted by Uli Köhler in Allgemein

How to check if NodeJS is run by root

Use this function to determine if the current NodeJS process is being run by the root user:

function isCurrentUserRoot() {
    return process.getuid() == 0; // UID 0 is always root
}

Usage example:

if(isCurrentUserRoot()) {
    // TODO Your code for root user goes here
} else {
    // TODO Your code for NON-root user goes here!
}

This works since the UID of the root user is always 0 on Linux/Unix systems.

Note: You don’t need to

const process = require("process");

since the process object is automatically imported in any NodeJS environment

Posted by Uli Köhler in Javascript, NodeJS

Should you use I2C or SPI devices for your project?

It depends on what you want to achieve. 

Note that some devices are only available as either I2C or SPI device and quite often the best solution is to use both I2C and SPI if you have enough pins to spare on your microcontroller. Many designers assume the bus is the primary selection criterion for part search, but often a part’s performance and price are more important than the interface it uses!

Advantages of I2C:

  • I2C uses only two pins (as opposed to SPIs three pins plus one address pin for each device for most configurations)
  • I2C can support multiple devices with the same two pins if each of them has a different address.
  • The I2C standard defines how to read and write to/from a specific address, hence you don’t need to check the datasheet as thoroughly for every individual device. Note that many complex devices extend this standard to a certain extent, e.g. larger EEPROMs have multiple I2C addresses to allow accessing the entire memory.
  • I2C has fewer parameters to configure since the speeds and other attributes are fixed in the standard (100 kbit/s, 400 kbit/s or 1 MBit/s = FM+). Hence you don’t need to worry about SPI modes or configuring the SPI speed correctly for any device.
  • The by-default slower speed of I2C makes it more likely for I2C to work over longer traces / wires out-of-the box. SPI is more prone to experiencing intermittent issues if operating at or near the phyiscal speed limit of a PCB design. With I2C it’s typically safe to assume it will work over 50cm long traces (for the standard 400 kbit/s speed). The only reason for I2C working over longer traces than SPI is it’s slower specified speed. SPI will work just as well if you set the speed sufficiently low.

Advantages of SPI:

  • SPI is much faster. While I2C typically communicates at 400 kbit/s (in Fast mode plus = FM+: 1 Mbit/s), SPI communicates around 20-50 MBit/s (it varies depending on your configuration and what speed each part supports). However for many applications you don’t need as much speed.
  • With SPI you don’t need to worry about assigning I2C addresses since each device has its own chip select pin. See our guide on assigning I2C addresses for more details.
  • SPI is more flexible in what data can be transferred since the communication message format is not standardized as much as in I2C.
  • Each SPI pin communicates only in a single direction. Hence SPI is much easier to galvanically isolate using optocouplers or similar devices. I2C requires special (and hence expensive) isolator devices like the TI ISO1541.
  • SPI is typically slighly easier to debug using an oscilloscope since you know what device the data comes from for every individual pin. With I2C, you need to have a look at the data sent to find out what device tries to send data.
  • I2C sometimes experiences issues due to locked-up devices if an I2C communication is interrupted mid-byte (e.g. by a microcontroller restart) and hence causes a device’s state machine to remain in a state mismatched to the I2C master’s state. Most designers do not anticipate this issue and do not include I2C lockup mitigation in their firmware. This means that often a complete power cycle of a system is required in order to remove the bus lockup. The issue occurs more often during firmware development when microcontroller reboots are more frequent due to reprogramming, crashes and breakpoints and in production tests it’s often hard to create the specific conditions that create a lockup in deployment. SPI can never lock up because it will reset once the chip select pin is deasserted.
  • SPI can often be used to control logic ICs like th 74HC series so you don’t need separate pins to control logic devices. Note that whether this works

In what cases do you need to use a different interface (other than I2C or SPI)?

  • Long cables / lines. SPI and I2C operate reliably up to a few tens of centimeters depending on your layout and the speed of the bus. In case you have longer traces, you might need to slow down the bus (especially in the case of SPI). The main issue here is clock skew: A change of the voltage level on the clock pin takes too long to arrive at the device, hence the device “answers too late” (additionally the response from the device takes too long to arrive at the master). If you want to check if this is the case for your design, just slow down your bus to extremely slow speeds (like 20 kbit/s). If the issue persists, you don’t have an issue with clock skew. You can also debug that using an oscilloscope.
  • Electromagnetic noise from the environment, e.g. large motors starting in the vincinity of your device. Since both I2C do not use differential lines
  • Multi-master communication: I2C and SPI typically only communicate when a (pre-defined) master initiates the communication. In most configurations, it’s not possible for a device to initiate communication using only the I2C or SPI bus, so you often need to poll a device many times per second to check if a status has changed. I2C and SPI do not have a built-in collision detection mechanism that detects if multiple devices are trying to write to the bus at the same time.
  • Asynchronous communications: If you want to send/receive data independently from each other, you should use UART, Ethernet or a similar bus that has completely independent send/receive subsystems that do not depend on a common clock.
  • Very fast communication: FPGAs, fast ADCs/DACs, flash memories etc might use e.g.:
    • QSPI: SPI with multiple data pins (typically 4 data pins – hence the name quad SPI).
    • LVDS: High-speed differential bus system
    • JESD204B: Ultra-high speed (multi-gigabit) data link system for ICs. Don’t touch if you don’t know what you’re doing!
    • Parallel: Just a lot of pins without anything too special.

What other interfaces should you consider if communicating outside the circuit board?

  • RS485/RS422: Robust high-speed industrial bus (physical layer only, used in systems like Profibus). Fast (15 MBit/s). RS422 is like UART but differential. RS485 operates receive/transmit on the same differential line.
  • RS232: Old industrial bus system, not differential, requires +-15V to communicate (and hence requires a special IC to generate these voltage levels). Somewhat robust, but not as robust as it seems. Slow (0.1 MBit/s typical, but typically works up to ~0.5 Mbit/s under most circumstances, if all devices support it), does not support long cables, especially not in electromagnetically noisy environments.
  • LVDS: High speed, but not over long distances (a few meters at most). Differential and somewhat robust, but not that suitable for noisy industrial environments if used with long-ish cables. Requires knowledge of high-speed circuit design to do correctly.
  • Ethernet: Very complex to do right from a PCB standpoint (you need a special PHY = physial layer transceiver IC and a microcontroller that supports the MAC layer that generates the Ethernet messages = frames) but very fast (100 Mbit/s max for most microcontrollers, but FPGAs, processors etc) and very robust (differential, 32-bit CRC checksums). Very convenient to interface to computers ; switchable. Supports TCP/IP. PHYs and other hardware are very cheap since they are produced in large quantities. Note: PHYs are rather complex and if you don’t understand some basics of high-speed circuit design and read all the datasheets very carefully, you have a high likelyhood of your design not working – which is often complex to debug especially if you don’t have a multichannel Oscilloscope capable of at least 50 MHz of Bandwidth and some experience in high speed circuit debugging. The interface from the PHY to your microcontroller ist typically RMII for 100 MBit/s Ethernet, additionally you need and SMC interface which is like I2C in order to configure the PHY.
  • USB: Fast, but only goes up to 480 MBit/s under ideal circumstances if your IC actually supports it (many need an external transceiver = USB HS PHY). Most microcontrollers support only up to 12 MBit/s IC . Complex, unless you use ready-to-use ICs. Debugging is pretty hard in practice since USB firmware can fail in many non-straightforward ways (unless, again, for ready-to-use ICs that only perform USB-to-X-conversion). I recommend to use an USB-to-UART-converter IC like the FT230XQ. USB on microcontrollers sometimes tends not to work with some computers unless done completely right, so if possible to try to use a professionally-made USB example that has been tested with a variety of devices and try to not modify it too much. Or just use a ready-to-use converter IC in the first place.
  • UART: Useful for debugging but not robust. UART-to-USB ICs are widely available (I currently recommend the FT230XQ) to interface to computers. Somewhat fast (can do up to ~12 MBit/s with many modern microcontrollers, but it’s not always feasible to use it at that speed, especially not with long traces / cables due to signal degradation). USB-to-UART adapters for debugging are readily available & very cheap in China.
  • CAN: Used mostly in automotive systems. Not very fast (1 MBit/s) but has a priority system that allows high-priority message to be sent during transmission of lower-priority messages (which are interrupted and re-sent later). Differential and quite robust. Mostly easy to use on the IC level but hard to do completely right on the firmware level (there are a lot of non-straightforward configuration options).
Posted by Uli Köhler in Electronics

How to visualize I2C first byte structure in TikZ

In our previous post on How many bits does an I2C address have? we showed this diagram:

This diagram was made using TikZ using the following source code:

% I2C first byte diagram
% Author: Uli Koehler (https://techoverflow.net)
\documentclass[tikz, border=1mm]{standalone}
\usetikzlibrary{positioning, decorations.pathreplacing, arrows.meta}

% Source: https://tex.stackexchange.com/a/24133/45450
\makeatletter
\newcommand*{\textoverline}[1]{$\overline{\hbox{#1}}\m@th$}
\makeatother

\begin{document}
\begin{tikzpicture}
\node (A6) [draw, minimum height=7mm, minimum width=10mm] {$A_6$};
\node (A5) [draw, minimum height=7mm, minimum width=10mm, right=0cm of A6] {$A_5$};
\node (A4) [draw, minimum height=7mm, minimum width=10mm, right=0cm of A5] {$A_4$};
\node (A3) [draw, minimum height=7mm, minimum width=10mm, right=0cm of A4] {$A_3$};
\node (A2) [draw, minimum height=7mm, minimum width=10mm, right=0cm of A3] {$A_2$};
\node (A1) [draw, minimum height=7mm, minimum width=10mm, right=0cm of A2] {$A_1$};
\node (A0) [draw, minimum height=7mm, minimum width=10mm, right=0cm of A1] {$A_0$};
\node (RW) [draw, minimum height=7mm, minimum width=10mm, right=0cm of A0] {R/\textoverline{W}};

\coordinate[above left=0mm of A6] (AddrLeft);
\coordinate[above right=0mm of A0] (AddrRight);

\coordinate[below left=0mm of RW] (RWLeft);
\coordinate[below right=0mm of RW] (RWRight);

\draw[decorate,decoration={brace,amplitude=10pt}, minimum height=9mm]
    (AddrLeft) -- (AddrRight)
    node[anchor=south,midway,above=8pt] {I2C address};

\draw[decorate,decoration={brace,amplitude=8pt}, minimum height=6mm]
    (RWRight) -- (RWLeft)
    node[anchor=north,midway,below=8pt] {Read/\textoverline{Write} bit};

\end{tikzpicture}
\end{document}

and compiled to SVG using this Makefile – for details, see How to export TikZ graphics as SVG:

%.pdf: %.tex
    pdflatex $<

%.svg: %.pdf
    pdf2svg $< $@

all: I2CFirstByte.svg

 

Posted by Uli Köhler in LaTeX

How many bits does an I2C address have?

I2C addresses always have 7 bits.

This is because the first byte in any I2C communication consists of the 7 bit address plus the Read/Write bit. The Read/Write bit is 0 if the I2C master requests to write to a specific address and it’s 1 if the I2C master requests to read from an address.

The first byte of the I2C communication looks like this:

Interested in how we made this diagram? Check out our post on How to visualize I2C first byte structure in TikZ

Posted by Uli Köhler in Electronics

How to export TikZ graphics as SVG

If you have a TikZ graphic, you can use the LaTeX standalone package to make the page fit the content:

% Minimal TikZ standalone example
\documentclass[tikz, border=1mm]{standalone}

\begin{document}
\begin{tikzpicture}
\draw (0,0) node [] {My text};
\end{tikzpicture}
\end{document}

Assuming you have saved that file as MyDiagram.tex, you can convert it to a PDF and subsequently convert that PDF to a SVG using

pdflatex MyDiagram.tex
pdf2svg MyDiagram.pdf MyDiagram.svg

which will generate this SVG:

Note that the 1mm border around the content is intentional and recommended for most usecases. The background is transparent by default (but has been set to white in HTML on this blogpost to illustrate the extent of the SVG).

You can also use this Makefile template:

%.pdf: %.tex
    pdflatex $<

%.svg: %.pdf
    pdf2svg $< $@

all: MyDiagram.svg

which allows you to automatically run the commands for one or multiple TeX files.

Posted by Uli Köhler in LaTeX

Minimal TikZ standalone example

This example contains a simple TikZ graphic using the standalone package, i.e. it will be exported to a PDF/DVI where the page just fits the content:

\documentclass[tikz, border=1mm]{standalone}

\begin{document}
\begin{tikzpicture}
\draw (0,0) node [draw=black] {My text};
\end{tikzpicture}
\end{document}

The result looks like this:

Assuming the file is named MyDiagram.tex, you can compile it to PDF using

pdflatex MyDiagram.tex

Note that our example contains a 1mm border by default since this seems to be more suitable for the average usecase than having no border at all. In order to change that, you just need to modify the first line, e.g.

\documentclass[tikz]{standalone} % No border

or

\documentclass[tikz]{standalone, border=5mm} % 5mm border
Posted by Uli Köhler in LaTeX

How to add box / border around node in TikZ

In TikZ, if you have a node like

\draw (0,0) node [] {My text};

you can add a border around it by using the draw=... attribute for the node:

\draw (0,0) node [draw] {My text};

You can also tell TikZ to draw it in blue:

\draw (0,0) node [draw=blue] {My text};

or tell it to draw it dashed:

\draw (0,0) node [draw, dashed] {My text};

You can also make the border thin or thick:

\draw (0,0) node [draw, very thin] {Very thin border};
\draw (0,-1) node [draw, thin] {Thin border};
\draw (0,-2) node [draw, semithick] {Semithick border};
\draw (0,-3) node [draw, thick] {Thick border};
\draw (0,-4) node [draw, very thick] {Very thick border};
\draw (0,-5) node [draw, ultra thick] {Ultra thick border};

Posted by Uli Köhler in LaTeX

How to fix vpnc-disconnect ‘no vpnc found running’

Problem:

You want to disconnect from your vpnc VPN using vpnc-disconnect, but you see this error message:

Solution:

vpnc-disconnect doesn’t know the difference between no vpnc is running and permission denied when trying to stop the vpnc daemon.

In most cases, using sudo:

sudo vpnc-disconnect

will fix the issue.

If this does not resolve the issue for you, check if you used the --pid-file argument to vpnc. vpnc-disconnect always assumes that the PID file is /var/run/vpnc.pid.

Alternatively, check out our alternate methods of stopping vpnc.

Posted by Uli Köhler in Allgemein

How to remove all docker networks

In order to remove all docker network interfaces (including all the bridge interfaces), run

docker network rm $(docker network ls -q)

in your favourite shell.

This will attempt to remove some pre-defined networks that can’t be removed and hence show you these error messages:

Error response from daemon: bridge is a pre-defined network and cannot be removed
Error response from daemon: host is a pre-defined network and cannot be removed
Error response from daemon: none is a pre-defined network and cannot be removed

You can safely ignore those error messages.

Posted by Uli Köhler in Container, Docker