Elasticsearch Python minimal index() / insert example

This minimal example inserts a single document into Elasticsearch running at http://localhost:9200:

#!/usr/bin/env python3
from elasticsearch import Elasticsearch

es = Elasticsearch()
es.index(index="test-index", id=1, body={"test": 123})

 

Posted by Uli Köhler in Databases, ElasticSearch, Python

How to use Cloudflare API key instead of token in Python Cloudflare API

Problem:

You want to access Cloudflare using the Cloudflare Python API like this:

#!/usr/bin/env python3
import CloudFlare
cf = CloudFlare.CloudFlare(
    email="[email protected]",
    token="Oochee3_aucho0aiTahc8caVuak6Que_N_Aegi9o"
)
# ...

but when you try to use the key=... argument like this:

cf = CloudFlare.CloudFlare(
    email="[email protected]",
    key="Oochee3_aucho0aiTahc8caVuak6Que_N_Aegi9o"
)

you see this error message:

Traceback (most recent call last):
  File "run.py", line 4, in <module>
    cf = CloudFlare.CloudFlare(
TypeError: __init__() got an unexpected keyword argument 'key'

Solution:

Just use the key in the token=... argument like this:

cf = CloudFlare.CloudFlare(
    email="[email protected]",
    token="[YOUR API KEY]"
)

This usage is officially documented in the README section of the Cloudflare API.

Posted by Uli Köhler in Networking, Python

How to fix Python Cloudflare CloudFlare.exceptions.CloudFlareAPIError: no token defined

Problem:

You want to run a program using the Cloudflare API, e.g. this example code:

#!/usr/bin/env python3
import CloudFlare

cf = CloudFlare.CloudFlare({
    "email": "[email protected]",
    "token": "Oochee3_aucho0aiTahc8caVuak6Que_N_Aegi9o" 
})
zones = cf.zones.get()
for zone in zones:
    zone_id = zone['id']
    zone_name = zone['name']
    print(zone_id, zone_name)

But when trying to run it, you see the following error message:

Traceback (most recent call last):
  File "run.py", line 8, in 
    zones = cf.zones.get()
  File "/usr/local/lib/python3.8/dist-packages/CloudFlare/cloudflare.py", line 672, in get
    return self._base.call_with_auth('GET', self._parts,
  File "/usr/local/lib/python3.8/dist-packages/CloudFlare/cloudflare.py", line 117, in call_with_auth
    self._AddAuthHeaders(headers, method)
  File "/usr/local/lib/python3.8/dist-packages/CloudFlare/cloudflare.py", line 90, in _AddAuthHeaders
    raise CloudFlareAPIError(0, 'no token defined')
CloudFlare.exceptions.CloudFlareAPIError: no token defined

Solution:

You are using the wrong syntax to give arguments to CloudFlare.CloudFlare(), use email=… and token=… arguments directly instead of using a dict!

cf = CloudFlare.CloudFlare(
    email="[email protected]",
    token="Oochee3_aucho0aiTahc8caVuak6Que_N_Aegi9o"
)

Note that you can’t do all operations with all tokens and if you perform an operation that is not possible with your token, you’ll see an error message like CloudFlare.exceptions.CloudFlareAPIError: Invalid request headers

Posted by Uli Köhler in Networking, Python

How to fix Python Cloudflare CloudFlare.exceptions.CloudFlareAPIError: no email and no token defined

Problem:

You want to run a program using the Cloudflare API, e.g. this example code:

#!/usr/bin/env python3
import CloudFlare

cf = CloudFlare.CloudFlare()
zones = cf.zones.get()
for zone in zones:
    zone_id = zone['id']
    zone_name = zone['name']
    print(zone_id, zone_name)

But when trying to run it, you see the following error message:

Traceback (most recent call last):
  File "test-cloudflare-api.py", line 5, in <module>
    zones = cf.zones.get()
  File "/usr/local/lib/python3.8/dist-packages/CloudFlare/cloudflare.py", line 672, in get
    return self._base.call_with_auth('GET', self._parts,
  File "/usr/local/lib/python3.8/dist-packages/CloudFlare/cloudflare.py", line 117, in call_with_auth
    self._AddAuthHeaders(headers, method)
  File "/usr/local/lib/python3.8/dist-packages/CloudFlare/cloudflare.py", line 88, in _AddAuthHeaders
    raise CloudFlareAPIError(0, 'no email and no token defined')
CloudFlare.exceptions.CloudFlareAPIError: no email and no token defined

Solution:

The Cloudflare API is missing the credentials you use to login. The easiest way to call the API with credentials is to initialize CloudFlare.CloudFlare() with the email and token as arguments

cf = CloudFlare.CloudFlare(
    email="[email protected]",
    token="Oochee3_aucho0aiTahc8caVuak6Que_N_Aegi9o"
)

Note that you can’t do all operations with all tokens and if you perform an operation that is not possible with your token, you’ll see an error message like CloudFlare.exceptions.CloudFlareAPIError: Invalid request headers

Posted by Uli Köhler in Networking, Python

Simple Elasticsearch setup with docker-compose

The following docker-compose.yml is a simple starting point for using ElasticSearch within a docker-based setup:

version: '2.2'
services:
    elasticsearch1:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.13.4
        container_name: elasticsearch1
        environment:
            - cluster.name=docker-cluster
            - node.name=elasticsearch1
            - cluster.initial_master_nodes=elasticsearch1
            - bootstrap.memory_lock=true
            - http.cors.allow-origin=http://localhost:1358,http://127.0.0.1:1358
            - http.cors.enabled=true
            - http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
            - http.cors.allow-credentials=true
            - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        ulimits:
            memlock:
                soft: -1
                hard: -1
        volumes:
            - ./esdata1:/usr/share/elasticsearch/data
        ports:
            - 9200:9200
    dejavu:
        image: appbaseio/dejavu
        container_name: dejavu
        ports:
            - 1358:1358

Now create the esdata1 directory with the correct permissions:

sudo mkdir esdata1
sudo chown -R 1000:1000 esdata1

We also need to configure the vm.max_map_count sysctl parameter:

echo -e "\nvm.max_map_count=524288\n" | sudo tee -a /etc/sysctl.conf && sudo sysctl -w vm.max_map_count=524288

 

I recommend to place it in /opt/elasticsearch, but you can place wherever you like.

If you want to autostart it on boot, see Create a systemd service for your docker-compose project in 10 seconds or just use this snippet from said post:

curl -fsSL https://techoverflow.net/scripts/create-docker-compose-service.sh | sudo bash /dev/stdin

This will create a systemd service named elasticsearch (if your directory is named elasticsearch like /opt/elasticsearch) and enable and start it immediately. Hence you can restart using

sudo systemctl restart elasticsearch

and view the logs using

sudo journalctl -xfu elasticsearch

For more complex setup involving more than one node, see our previous post on ElasticSearch docker-compose.yml and systemd service generator

Posted by Uli Köhler in Container, Databases, Docker, ElasticSearch

How to filter OpenStreetmap nodes by tag from .osm.pbf file using Python & Osmium

In our previous post Minimal example how to read .osm.pbf file using Python & osmium we investigated how to read a .osm.pbf file and count all nodes, ways and relations.

Today, we’ll investigate how to filter for specific nodes by tag using osmium working on .osm.pbf files. In this example, we’ll filter for power: tower and count how many nodes we can find

#!/usr/bin/env python3
import osmium as osm

class FindPowerTowerHandler(osm.SimpleHandler):
    def __init__(self):
        osm.SimpleHandler.__init__(self)
        self.count = 0

    def node(self, node):
        if node.tags.get("power") == "tower":
            self.count += 1

osmhandler = FindPowerTowerHandler()
osmhandler.apply_file("germany-latest.osm.pbf")

print(f'Number of power=tower nodes: {osmhandler.node_count}')

Also see How to filter OpenStreetmap ways by tag from .osm.pbf file using Python & Osmium

Posted by Uli Köhler in Geography, OpenStreetMap, Python

Minimal leaflet.js example with Stamen terrain tiles

This minimal self-contained HTML example can serve as a good starting point for your own leaflet application. Using Stamen Terrain tiles not only provides a nice view of the physical geography but has the added advantage of not requiring any API key.

<html>
    <head>
        <link rel="stylesheet" href="https://unpkg.com/[email protected]/dist/leaflet.css"
        integrity="sha512-xodZBNTC5n17Xt2atTPuE1HxjVMSvLVW9ocqUKLsCC5CXdbqCmblAshOMAS6/keqq/sMZMZ19scR4PsZChSR7A=="
        crossorigin=""/>
        <script src="https://unpkg.com/[email protected]/dist/leaflet.js"
        integrity="sha512-XQoYMqMTK8LvdxXYG3nZ448hOEQiglfqkJs1NOQV44cWnUrBc8PkAOcXy20w0vlaXaVUearIOBhiXZ5V3ynxwA=="
        crossorigin=""></script>
        <script type="text/javascript" src="https://stamen-maps.a.ssl.fastly.net/js/tile.stamen.js?v1.3.0"></script>
        <style>
            #mymap { height: 100%; }
        </style>
    </head>
    <body>
        <div id="mymap"></div>
        <script type="text/javascript">
            var layer = new L.StamenTileLayer("terrain");
            var map = new L.Map("mymap", {
                /* Center: Munich, Germany */
                center: new L.LatLng(48.1, 11.5),
                /* Show most of western Europe */
                zoom: 6
            });
            map.addLayer(layer);
        </script>
    </body>
</html>

Look and feel of the example:

Posted by Uli Köhler in HTML, Javascript, Leaflet

How to read single value from XLS(X) using pandas

pandas can be used conveniently to read a table of values from Excel. When extracting data from real-life Excel sheets, there are often metadata fields which are not structured as a table readable by pandas.

Reading the pandas docs, it is not obvious how we can extract the value of a single cell (without any associated headers) with a fixed position, for example:

In this example, we want to extract cell C3, that is we want to end up with a string of value Test value #123. Since there is no clear table structure in this excel sheet (and other cells might contain other values we are not interested in – for example, headlines or headers), we don’t want to have a pd.DataFrame but simply a string.

This is how you can do it:

 

def read_value_from_excel(filename, column="C", row=3):
    """Read a single cell value from an Excel file"""
    return pd.read_excel(filename, skiprows=row - 1, usecols=column, nrows=1, header=None, names=["Value"]).iloc[0]["Value"]

# Example usage
read_value_from_excel(“Test.xlsx”, “C”, 3) # Prints
Let’s explain the parameters we’re using as arguments to pd.read_excel():

  • Test.xlsx: The filename of the file you want to read
  • skiprows=2: Row number minus one, so the desired row is the first we read
  • usecols="C": Which columns we’re interested in – only one!
  • nrows=1: Read only a single row
  • header=None: Do not assume that the first row we read is a header row
  • names=["Value"]: Set the name for the single column to Value
  • .iloc[0]: From the resulting pd.DataFrame, get the first row ([0]) by index (iloc)
  • ["Value"] From the resulting row, extract the "Value" column – which is the only column available.

In my opinion, using pandas is the best way of extracting for most real-world usecases (i.e. more focused on development speed than on execution speed) because not only does it provide automatic engine selection for .xls and .xlsx files, it’s also present on most Data Science setups anyway and provides a standardized API.

Posted by Uli Köhler in pandas, Python

How to fix pandas pd.read_excel() error XLRDError: Excel xlsx file; not supported

Problem:

When trying to read an .xlsx file using pandas pd.read_excel() you see this error message:

XLRDError: Excel xlsx file; not supported

Solution:

The xlrd library only supports .xls files, not .xlsx files. In order to make pandas able to read .xlsx files, install openpyxl:

sudo pip3 install openpyxl

After that, retry running your script (if you are running a Jupyter Notebook, be sure to restart the notebook to reload pandas!).

If the error still persists, you have two choices:

Choice 1 (preferred): Update pandas

Pandas 1.1.3 doesn’t automatically select the correct XLSX reader engine, but pandas 1.3.1 does:

sudo pip3 install --upgrade pandas

If you are running a Jupyter Notebook, be sure to restart the notebook to load the updated pandas version!

Choice 2: Explicitly set the engine in pd.read_excel()

Add engine='openpyxl' to your pd.read_excel() command, for example:

pd.read_excel('my.xlsx', engine='openpyxl')

 

Posted by Uli Köhler in pandas, Python

Who is client and who is server in Wireguard?

Wireguard doesn’t really use the concept of client and server the same way OpenVPN does. A wireguard interface does not have a fixed role as client or server – think about it like this:

  • A wireguard connection is a link between two peers
  • One wireguard interface can host one or many connections

For a single connection:

  • connection can be considered a client if it knows a fixed endpoint (IP address or hostname) to connect to, i.e. if you have Endpoint set in your wireguard config like this:
    Endpoint = vpn.mydomain.com:31265

    client will take the initiative and send packets to the server without having received any packet from the server beforehand – just like in classical VPNs.

  • connection can be considered a server if it doesn’t have an Endpoint set to connect to. A server will learn which IP address to send packets to once a client has completed the handshake. If a client IP address changes, the server will learn the new IP address as soon as it receives a validated packet from the client.

Most real-world wireguard connections have one client and one server. There are exceptions to this, namely if both endpoints have a static IP address or fixed host name, so both wireguard instances always know which IP address or hostname to send packets to.

Posted by Uli Köhler in Networking, VPN, Wireguard

What to look for in “wg show” output?

This is an example wg show output:

interface: MyVPN
  public key: xJ+A//t9RbOU4ISIr61tsZwc8SPLbLONXhknnU1QvBQ=
  private key: (hidden)
  listening port: 12073

peer: xgmml6wPoe9auL5oGhqScQXLByfrI/1xq3sOJzYaNhE=
  endpoint: 77.55.81.22:23711
  allowed ips: 10.178.212.1/32, 10.39.24.0/24
  latest handshake: 37 seconds ago
  transfer: 948 B received, 1.40 KiB sent
  persistent keepalive: every 30 seconds

This is what I look for:

  • Is the desired wireguard interface present? If not, this indicates that either the computer doesn’t even try to start the interface (e.g. because autostart is not enabled) or starting it fails, for example because the route is already defined
  • Are the desired peers listed? If not, this is always a configuration error
  • Is persistent keepalive enabled? Without persistent keepalive, you will not be able to properly debug Wireguard because no packets will be sent unless some traffic is going through the interface. Therefore, I strongly recommend to always enable persistent keepalive even if you plan to disable it later!
  • Is latest handshake listed and recent? Not being able to handshake with a remote peer typically indic-ates either a network problem or a configuration problem, but in some cases it’s also a system-related problem:
    • System problems: Is wireguard interface on the local & remote side up & configured?
    • Networking problems: Port not forwarded to destination machine, TCP port instead of UDP port forwarded, local or remote internet access is firewalled, incorrect port given, incorrect IP address or hostname given, DynDNS hostname not updated, Wireguard tries to access IPv6 address but only IPv4 port is forwarded properly (check using host)
    • Wireguard configuration problem: Does the remote peer use the correct private key that matches the public key in the local configuration? Does the remote configuration have listed the local public key as peer at all? Does the local configuration have the correct private key that matches the public key listed in the remote config? Does the peer public key match the endpoint (if specified) or maybe the key doesn’t match the endpoint?
  • transfer should show >0 bytes received and sent! This is typically equivalent to the latest handshake debugging method. Bytes being sent but no bytes being received typically indicates that the Wireguard interface is trying to perform an handshake but does not get any reply back.

Also see my WireguardConfig project which makes this kind of configuration much easier

Posted by Uli Köhler in Networking, VPN, Wireguard

Install & autostart Wireguard config on Ubuntu or Debian

We will assume that you already have a wireguard config file, e.g. MyVPN.conf

  1. Copy MyVPN.conf to /etc/wireguard/MyVPN.conf:
    sudo cp MyVPN.conf /etc/wireguard/MyVPN.conf
  2. Start & enable (i.e. autostart) service:
    sudo systemctl enable --now wg-quick@MyVPN
  3. Check if it works using
    sudo wg show

Example wg-show output

interface: MyVPN
  public key: xJ+A//t9RbOU4ISIr61tsZwc8SPLbLONXhknnU1QvBQ=
  private key: (hidden)
  listening port: 12073

peer: xgmml6wPoe9auL5oGhqScQXLByfrI/1xq3sOJzYaNhE=
  endpoint: 77.55.81.22:23711
  allowed ips: 10.178.212.1/32, 10.39.24.0/24
  latest handshake: 37 seconds ago
  transfer: 948 B received, 1.40 KiB sent
  persistent keepalive: every 30 seconds

 

Posted by Uli Köhler in Networking, VPN, Wireguard

What 1500pF 1kV capacitor should you use for Ethernet termination?

Nowadays you can use cheap ceramic capacitors for Ethernet termination. While they are not self-healing like foil capacitors, they work fine for all but the most demanding applications. Also, it’s typically much cheaper to assemble SMD capacitors as opposed to through-hole foil types since the SMD types can be more easily picked & placed by machines.

I recommend to use the Yageo CC1206KKX7RCBB152 because it’s cheap (0,10€ @100pcs) and readily available at every major distributor.

Furthermore, it has a X7R ceramic, meaning that its capacitance doesn’t change too much with temperatur (not as much as, for example, Y5V ceramics). Note that X5R and X7R types have a capacity that depends on the voltage being applied, so if you have an application that is really sensitive to RF noise, keep in mind that if a significant DC voltage is applied to the capacitor, its capacitance will drop by up to tens of percents – leading to impromper Ethernet termination and hence more EMI.

Additionally, keep in mind that ceramic capacitors are somewhat susceptiple to mechanical flexing of the PCB. This is especially the case if you have a very thin (or even rigid-flex) PCB, or if high mechanical loads (either static loads or vibrations) are applied directly to your PCB. In that case, consider buying e.g. a slightly more expensive, “flexible termination” type capacitor.

Posted by Uli Köhler in Compliance, Electronics, EMI

What are “MIB counters” in Ethernet?

In some Ethernet switch or router datasheets you see sentences like

Gathers 34 MIB counters per port

MIB means Management Information Base, that is information that can be used to manage a networking system.

A typical set of MIB counters is:

  • Number of packets received (for each port)
  • Number of packets sent (for each port)
  • Number of bytes transmitted (for each port)
  • Number of bytes received (for each port)
  • Number of packets with faulty CRC received (for each port)
  • Number of unicast packets received (for each port)
  • Number of broadcast packets received (for each port)
  • Number of small packets (<64 bytes) (for each port)
  • Number of small-ish packets (64-127 bytes) (for each port)
  • Number of small-to-medium packets (128-255 bytes) (for each port)
  • Number of medium packets (256-511 bytes) (for each port)
  • Number of medium-to-large packets (512-1023 bytes) (for each port)
  • Number of large packets (1024-1522 bytes) (for each port)
  • Number of CSMA/CD collisions during transmission (for each port)
  • Number of times a single packet had multiple CSMA/CD collisions (for each port)
  • Number of deferred packets (packet had a collision but will be tried again (for each port)
  • Number of packets dropped because buffer is full (for each port where the packet is received)

These can be used to compute other parameters such as the current throughput. For example, to calculate the transmit throughput, use

  • Query number of bytes transmitted, store in variable A
  • Wait 1 second
  • Query number of bytes transmitted again, store in variable B
  • Number of bytes transmitted in that second is (B-A)
  • Throughput is (B-A)/1s
Posted by Uli Köhler in Electronics, Networking

What to do if your RTC is not accurate enough?

If your RTC drifts too much over time, here are some solutions you need to consider – but keep in mind that not all variants

Did you select the correct clock source?

Many RTCs and controllers have multiple clock sources – typically, one internal RC oscillator and one external crystal oscillator. The RC oscillator is typically much more inaccurate, hence you should always select. You should absolutely double-check if you have set the clock source correctly – for example by checking with an oscilloscope if the crystal is oscillating. Note that this is not 100% reliable since your oscilloscope probe is loading the crystal with its additional capacitance and might change the frequency significantly – and in some cases the extra capacitance will cause the crystal to stop oscillating. In practice, it works most of the time though, so it’s always worth a try – just don’t take the measured frequency too seriously.

Did you select the correct load capacitance?

In our post on How to compute crystal load capacitors using Python we showed an easy method of how to compute the correct load capacitor value. If in doubt, redo the calculation – often, people make the mistake of looking for the load capacitance in the datasheet and assuming that this is the value of the capacitors to attach to the crystal.

Additionally, since the load capacitance calculation involves some estimation of the board capacitance, you should tune the correct capacitance once you have a prototype board, i.e. selecting a slighly different load capacitor value to get the desired accuracy. See our post on How to tune your crystal oscillator to get the best possible frequency accuracy for further instructions.

Can you use a better crystal?

Often you can just spend a few cents more to get a crystal that has an higher accuracy and/or a lower temperature coefficient. Note that having a more accurate crystal is typica

Can you digitally tune the RTC?

Many modern RTCs feature a way of digitally tuning the exact frequency using register settings. Our post on How to tune your crystal oscillator to get the best possible frequency accuracy has a sections containing more information on how you can approach this. Note that digital tuning can not by itself compensate for temperature changes leading to your crystal frequency changing.

Can you automatically compensate for RTC drift using your microcontroller?

If your product is powered on at least a couple of times a year (i.e. it is not powered by a CR2032 or similar battery), you can use that timespan to recalibrate your RTC using digital tuning, or to manually tune the RTC by computing the number of hours since the last power on event and subtracting e.g. a few seconds for each hour to adjust for the RTC drift. This approach is most accurate if you have any way of comparing the current RTC frequency to another clock source (see below). Note that automatic compensation can only adjust for temperature changes leading to your crystal frequency changing if the MCU is running, the second clock source is running and if the second clock source

Can you synchronize to a second clock source?

Using your microcontroller, you can build a frequency counter in firmware that will compute that actual frequency of your RTC crystal. In order for this to work, you need to have a second clock source with a higher accuracy.

These are the most common alternate clock sources you can use for your board:

  • The high speed oscillator clocking your microcontroller. If power usage is not a major concern, I recommend using an oscillator, not a crystal, because you don’t have to tune load capacitors when using an oscillator, leading to higher effective accuracy in many real-world usecases – especially, if you don’t want to tune at all.
  • The power grid has a frequency of 50 Hz or 60 Hz depending on where you are in the world. It has an excellent long-term stability, but its short term stability is not that great. The people running the power grid are continously compensating the short term instabilities so that the. Note that if you want to use the power grid as a clock source, you have to take care of proper isolation and spacing since grid power can be extremely dangerous.
  • GPS is rather complex to implement but provides an extremely accurate clock source if you have GPS reception. Note that it typically won’t work indoors and it’s a very expensive yet very accurate way of clocking your device. Since it’s also rather power-intensive, it is often not a good option for battery powered products.
  • In Germany, the DCF77 RF transmitter sends a time signal every minute. I recommend to buy ready to use DCF77 modules like this one if at all possible
  • Some microcontrollers (for example, some STM32F0) can use a computer’s USB as clock source once they have established a USB connection. This does not work when using an external Serial-to-USB converter

If you can use neither of those clock sources, you could do some significant development to recover clock sources from sources like:

  • Ethernet
  • USB
  • CAN
  • UART
  • Some RF signal like Wifi, NFC

but as far as I know there are no ready-to-use modules except those listed above. Note that there are many circumstances under which those clock source just won’t work reliably, and implementing it will be rather difficult, so I don’t recommend to go down this path of tears.

Can you synchronize to the internet?

If your device has internet connectivity or connectivity to a network like Sigfox or LoRa, consider if you can just synchronize the time with the internet or just, for example, send a message to your device every day at 00:00 so that it can compensate for the drift. You could also continously adjust the RTC drift by digital calibration or automatic compensation in firmware.

Posted by Uli Köhler in Electronics

Beware of the STM32 LSI: Its tolerance is up to ±47%

The STM32 LSI oscillator might seem like an attractive choice for RTC, IWDG Watchdog etc – without external components and

But one fact is often overlooked: Since it is internally a RC oscillator, it has an extremely high tolerance.

By looking at the STM32F407 datasheet, for example, we can see that its tolerance is ±47%

In other words, the LSI clock can run half as slow or 1.5 times as fast as expected.

±47% is equivalent to ±470 000 ppm whereas any normal crystal has a tolerance of ±20 ppm.

More recent STM32 families like the STM32H747XI have improved LSI accuracy:

This amounts to a tolerance of ±1.875 % which is equivalent to ±18 750 ppm – still orders of magnitude more than any crystal or even ceramic resonator.

Can you tune out the difference by RTC digital tuning?

The STM32 digital tuning only has a range of -487.1 ppm to +488.5 ppm – but even for the much more accurate STM32H747XI, you would need a tuning range of at least ±20 000 ppm in order to compensate for initial inaccuracies and temperature coefficient.

What can you do to get better accuracy?

Typically, I recommend to just use a crystal for the RTC – or use an external RTC altogether.

Regarding the IWDG, you have no choice but to use the LSI. Typically you can just select a longer reset interval to avoid unintended watchdog resets if your LSI is running much faster than the standard 32 kHz, or you can just reset the watchdog more often. If you reset your watchdog in an interrupt, you should consider using a higher priority interrupt – and do global interrupt disables less frequently and try to avoid having periods where interrupts are disabled globally for a long time continously.

 

Posted by Uli Köhler in Electronics, STM32

How to tune your crystal oscillator to get the best possible frequency accuracy

In our previous post How to compute crystal load capacitors using Python we investigated how to use UliEngineering to compute the appropriate load capacitor for your crystal oscillator.

Once you have manufactured your board, you should go one step further and tune your crystal oscillator.

Note that in this post we’re not talking about specialized compensated or heated crystals like TCXOs, DCXOs or OCXOs, but about your normal crystal which you use to clock your RTC, your microcontroller, your Ethernet PHY, your ADC, …

Also, this post is not made primarly for ppm hunters who desire to get below-±5ppm accuracy. While you certainly need to apply the techniques outlined here very carefully to get below your magic ppm number, they will likely not be sufficient.

Tuning the load capacitor

Note that for new PCBs, the stray PCB capacitance is just an educated guess – so your load capacitance will be slightly off.

First, you need to remember: Too low a load capacitance will result in higher frequency – too high a load capacitance will result in lower frequency.

I do not recommend to measure the PCB capacitance directly, because capacitances in the single picofarads are hard to measure accurately. Additionally, you can’t just measure them on a spare board without all the components (since the components, for example the case of the crystal, will affect the stray capacitance) but you also can’t just measure it on a fully populated board since the crystal itself will strongly influence your measurement.

Instead, just measure the  frequency of the crystal oscillator by measuring a reference clock output. The most appropriate instrument for that is a frequency counter with a sufficiently stable frequency source. Oscilloscopes typically don’t have the resolution required to measure a frequency down to the ppms. But if you have a microcontroller board with a sufficiently stable crystal.

Note that by attaching your probe to the crystal, you will load the crystal with additional capacitance from your probe, hence you won’t measure its actual frequency accurately. Do not probe your crystal directly but configure your RTC so that it generates a clock output on a separate pin and measure that.

Typically, your initially calculated load capacitor value will be within ±50% of the final value. Hence, you should buy approximately 5-20 standard values of capacitors within that range. Nowadays, buying 10 ceramic capacitors costs only around 0.15€ so just buying a buch of them is not really worth any consideration and you should just order them – or alternatively order a kit of capacitors. Note that standard capacitor values like 10pF or 6.2pF are often much cheaper than very specific values, so take care when selecting the values you buy.

For example, if you have a calculated load capacitor value of 7pF, you could order the following capacitors:

  • 3.5 pF
  • 4 pF
  • 5 pF
  • 6.2 pF
  • 7 pF
  • 8.2 pF
  • 10 pF

How accurately should you tune?

My recommendation on how accurately you should tune depends on whether exchanging components is the only type of tuning you can do or if your oscillator IC supports digital tuning. Since nowadays digital tuning is so easy and much more reproducible than changing components (due to tolerances etc)

If your RTC/controller does NOT support digital tuning: My recommendation is to tune no more than 2.5 times the specified accuracy of the crystal. So if your crystal has specified tolance of ±20ppm, in most applications you should tune to ±50ppm

If your controller supports digital tuning but not autotuning: My recommendation is to tune no more than 4 times the specified accuracy of the crystal. So if your crystal has specified tolance of ±20ppm, in most applications you should tune to ±80ppm

If your controller supports digital autotuning: My recommendation is to tune to Digital adjustment range / 1.5

In any case, remember that it’s a tradeoff between your time improving the product and slight inaccuracies that might

Digitally tuning the frequency

Many controllers and RTCs nowadays feature a circuit to digitally tune the frequency to up to sub-ppm levels using e.g. I2C access. Internally, this works by omitting or adding pulses to an internal clock every now and then, effectively decreasing or increasing the average frequency of the clock.

Note that you can’t just infinitely adjust using digital adjustment, a typical adjustment range will be ±100ppm to ±500ppm.

The most primitive variant is to just use the register where you enter how many PPMs up or down the frequency should be adjusted, do it once for your prototype and hope that your production run won’t be significantly different. In this case, you have little choice but to take your frequency counter, measure the frequency. Note that digital adjustment does not change the frequency of the crystal at all, but typically the reference output is digitally adjusted. Note however, that the adjustment might only happen every few minutes (read your RTC datasheet or reference manual for more info). The only way to reliably check if the change is effective is by letting the board run for a few days and observe how much it drifted compared to a reference clock.

A more advanced, albeit much harder to implement variant is to use a different (more accurate) reference clock source like an accurate oscillator or even OCXO, or a clock derived from USB (which some STM32 microcontrollers can do) or the power grid (which is not very accurate over short time spans but is very accurate over long time spans), implement a frequency counter in your MCU and automatically adjust the digital tuning parameters. The implementation of this highly depends on which MCU you are using and would exceed the scope of this post.

Posted by Uli Köhler in Electronics

How to compute crystal load capacitors using Python

The UliEngineering library provides a convenient way to compute which load capacitors are suitable for a given crystal oscillator circuit.

First, install UliEngineering using

sudo pip3 install UliEngineering

Now you need to find out the following parameters:

  • The load capacitance from the crystal’s datasheet. Typical values would be 7pF or 12.5pF or 20pF
  • The pin capacitance CPin from the datasheet of the IC your crystal is connected to (e.g. your Ethernet PHY or your RTC). Typical values would be between 0.5pF to 5pF
  • Estimate the stray capacitance CStray from the board (that is the capacitance of each crystal oscillator pin to the PCB ground). You typically just take an educated guess here, so here are some values to start at:
    •  Start at 3pF
    • If you have a 4+ layer board (as opposed to a two layer board), add 2pF (since the distance to the ground plane below is much smaller on 4 layer boards
    • If you have long traces >1cm (long traces to the crystal should be avoided at all costs!), add 4pF for each cm of trace beyond 1cm for 2-layer boards or add 6pF for each cm of trace beyond 1cm for 4+ layer boards

Now we’ll plug those values into UliEngineering and compute the load capacitance:

from UliEngineering.Electronics.Crystal import load_capacitors
from UliEngineering.EngineerIO import auto_print

auto_print(load_capacitors, cload="9 pF", cpin="1 pF", cstray="5 pF")

This will print 7.00 pF

If you just want to the the value and not an auto-formatted string, use load_capacitors() directly:

capacitor = load_capacitors(cload="9 pF", cpin="1 pF", cstray="5 pF")

In this example, we’ll get capacitor == 7e-12.

Note that 7pF means that you have to add a 7pF capacitor at each side of the crystal. I recommend to use only NP0/C0G capacitors here since X5R/X7R capacitors and other high-k-dielectric ceramic capacitors not only have a high temperature coefficient but also they are not as suitable for analog applications since their capacitance will change with the voltage being applied.

Note that we used a lot of guesswork in the PCB stray capacitance estimation above, so if you need high accuracy, you need to tune your crystal oscillator to work best with your specific board. See our followup post on How to tune your crystal oscillator to get the best possible frequency accuracy for further reading.

Posted by Uli Köhler in Electronics, Python

How to connect ESP32-WROOM-32 SENSOR_VP & SENSOR_VN pins?

If you are making a PCB using the ESP32-WROOM-32 module, you might be wondering how to connect theSENSOR_VP and SENSOR_VN pins (pins 4 & 5).

  • These pins are made to accurately measure differential low-voltage signals using the ESP32 12-bit ADC. If you want to measure a differential signal, connect SENSOR_VP to the positive voltage of your analog signal and connect SENSOR_VN to the negative voltage of your analog signal. Take care not to exceed the maximum voltage range of approx. 0..3.3V for the ESP32, else you will damage the chip!
  • These pins can be used as normal GPIOsSENSOR_VP is GPIO36 and SENSOR_VN is GPIO39however these are input-only, you can’t use them as output!
  • If you don’t need the pins, connect them to GND, or just leave them open (i.e. don’t connect them at all)

Source & further reading: ESP32-WROOM-32 reference manual

Posted by Uli Köhler in Electronics, ESP8266/ESP32

What is the SPI pinout of the ESP32 / ESP-WROOM-32?

When using the ESP32 as SPI master, you can use any pins for the SCLKMISOMOSI and CS signals, but using the following set of pins has minor advantages:

SPI pin nameESP32 pin (SPI2)ESP32 pin (SPI3)
CS155
SCLK1418
MISO1219
MOSI1323

If you use all of the pins for SPI2 or all of the pins for SPI3, using those pins is slightly faster, since the signals do not have to be routed through the GPIO matrix. This has the advantage of having a lower input delay (which is important at high speeds to avoid issues with MISO setup time) and that you can operate the SPI bus at 80 MHz (as opposed to 40 MHz with the GPIO matrix).

If you use ANY pin beside those listed above, ALL pins will be routed through the GPIO matrix – so use either all of these pins or ignore it altogether. Note that some pins are input-only or reserved for special functions, so they may not be used for some or all of the SPI signals.

Source & further reading: ESP32 SPI master driver documentation

Posted by Uli Köhler in Electronics, ESP8266/ESP32
This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Cookie settingsACCEPTPrivacy &amp; Cookies Policy