Python

LinuxCNC: How to find current position using Python

This will show the position in machine coordinates such as

#!/usr/bin/env python2.7
import linuxcnc

stat = linuxcnc.stat()
stat.poll()
print(stat.actual_position)

Example output:

(7.4023762662053105, 26.443582149595567, 297.289833343029, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0)

 

Posted by Uli Köhler in LinuxCNC, Python

LinuxCNC: Custom M-100 G-Code that logs time, parameters & filename to CSV

Place this file in e.g. linuxcnc/configs/myCNC/custom-mcode/M100

#!/usr/bin/env python2.7
# M100: write timestamp, parameters and gcode filename to linuxcnc/logM100.txt
import sys
import linuxcnc
from datetime import datetime
dt = datetime.now()

stat = linuxcnc.stat() # create a connection to the status channel
stat.poll()

with open("/home/cnc/linuxcnc/logM100.txt", "a") as outfile:
    outfile.write("{} | Args={} | Path={}\n".format(dt.isoformat(), ", ".join(sys.argv[1:]), stat.file))

and make executable using

chmod a+x ./linuxcnc/configs/myCNC/custom-mcodes/M100
Posted by Uli Köhler in LinuxCNC, Python

LinuxCNC: How to read current position in Python and log to CSV

Also see our post on how to just read the position especially if you don’t care about the CSV logging: LinuxCNC: How to find current position using Python

This script will log the position to CSV approximately every millisecond. The position will be logged in machine coordinates.

#!/usr/bin/env python2.7
import linuxcnc
import datetime
import time

stat = linuxcnc.stat()

with open("position-log.csv", "w") as outfile:
    while True:
       dt = datetime.datetime.now()
       stat.poll()
       x,y,z,a,b,c,u,v,w = stat.actual_position
       outfile.write("{},{:.4f},{:.4f},{:.4f}\n".format(dt.isoformat(), x, y, z))
       time.sleep(0.001)

 

Posted by Uli Köhler in LinuxCNC, Python

How to parse linuxcnc.var using Python

with open("/home/cnc/linuxcnc/configs/myCNC/linuxcnc.var") as infile:
    lines = infile.readlines()
    splitted = [line.strip().partition("\t") for line in lines]
    linuxCNCVar = {int(splitResult[0]): float(splitResult[-1]) for splitResult in splitted}

Example output for linuxCNCVar:

{5161: 0.0,
 5162: 0.0,
 5163: 0.0,
 5164: 0.0,
 5165: 0.0,
 5166: 0.0,
 5167: 0.0,
 5168: 0.0,
 5169: 0.0,
 5181: 0.0,
 5182: 0.0,
 5183: 0.0,
 5184: 0.0,
 5185: 0.0,
 5186: 0.0,
 5187: 0.0,
 5188: 0.0,
 5189: 0.0,
 5210: 0.0,
 5211: 0.0,
 5212: 0.0,
 5213: 0.0,
 5214: 0.0,
 5215: 0.0,
 5216: 0.0,
 5217: 0.0,
 5218: 0.0,
 5219: 0.0,
 5220: 1.0,
 5221: 7.40119,
 5222: 13.549249,
 5223: 298.476583,
 5224: 0.0,
 5225: 0.0,
 5226: 0.0,
 5227: 0.0,
 5228: 0.0,
 5229: 0.0,
 5230: 0.0,
 5241: 36.868548,
 5242: 141.752329,
 5243: 207.563262,
 5244: 0.0,
 5245: 0.0,
 5246: 0.0,
 5247: 0.0,
 5248: 0.0,
 5249: 0.0,
 5250: 0.0,
 5261: 36.868548,
 5262: 145.029329,
 5263: 207.563262,
 5264: 0.0,
 5265: 0.0,
 5266: 0.0,
 5267: 0.0,
 5268: 0.0,
 5269: 0.0,
 5270: 0.0,
 5281: 36.89201,
 5282: 144.834924,
 5283: 207.935105,
 5284: 0.0,
 5285: 0.0,
 5286: 0.0,
 5287: 0.0,
 5288: 0.0,
 5289: 0.0,
 5290: 0.0,
 5301: 0.0,
 5302: 0.0,
 5303: 0.0,
 5304: 0.0,
 5305: 0.0,
 5306: 0.0,
 5307: 0.0,
 5308: 0.0,
 5309: 0.0,
 5310: 0.0,
 5321: 0.0,
 5322: 0.0,
 5323: 0.0,
 5324: 0.0,
 5325: 0.0,
 5326: 0.0,
 5327: 0.0,
 5328: 0.0,
 5329: 0.0,
 5330: 0.0,
 5341: 0.0,
 5342: 0.0,
 5343: 0.0,
 5344: 0.0,
 5345: 0.0,
 5346: 0.0,
 5347: 0.0,
 5348: 0.0,
 5349: 0.0,
 5350: 0.0,
 5361: 0.0,
 5362: 0.0,
 5363: 0.0,
 5364: 0.0,
 5365: 0.0,
 5366: 0.0,
 5367: 0.0,
 5368: 0.0,
 5369: 0.0,
 5370: 0.0,
 5381: 0.0,
 5382: 0.0,
 5383: 0.0,
 5384: 0.0,
 5385: 0.0,
 5386: 0.0,
 5387: 0.0,
 5388: 0.0,
 5389: 0.0,
 5390: 0.0}

 

Posted by Uli Köhler in LinuxCNC, Python

How to fix tox AttributeError: module ‘virtualenv.create.via_global_ref.builtin.cpython.mac_os’ has no attribute ‘CPython2macOsArmFramework’

Problem:

While trying to run Python tests using tox, you see an error message like

GLOB sdist-make: /home/uli/dev/UliEngineering/setup.py
py37 create: /home/uli/dev/UliEngineering/.tox/py37
ERROR: InterpreterNotFound: python3.7
py38 create: /home/uli/dev/UliEngineering/.tox/py38
ERROR: invocation failed (exit code 1), logfile: /home/uli/dev/UliEngineering/.tox/py38/log/py38-0.log
========================================================================================= log start =========================================================================================
AttributeError: module 'virtualenv.create.via_global_ref.builtin.cpython.mac_os' has no attribute 'CPython2macOsArmFramework'

========================================================================================== log end ==========================================================================================
ERROR: InvocationError for command /usr/bin/python3 -m virtualenv --no-download --python /usr/bin/python3 py38 (exited with code 1)
__________________________________________________________________________________________ summary __________________________________________________________________________________________
ERROR:  py37: InterpreterNotFound: python3.7
ERROR:   py38: InvocationError for command /usr/bin/python3 -m virtualenv --no-download --python /usr/bin/python3 py38 (exited with code 1)

Solution:

This error occurs because two different and incompatible version of virtualenv are installed.

First, uninstall all versions using these commands:

First run as user (not as root) to uninstall any locally installed package in ~/.local:

pip3 uninstall virtualenv

Then, uninstall global packages

sudo pip3 uninstall virtualenv

Now uninstall the apt package on Debian/Ubuntu (uninstall using your package manager for other distros):

sudo apt purge python3-virtualenv

Now it’s time to install one version of virtualenv:

sudo pip3 install virtualenv

After that, you can try running tox again.

Posted by Uli Köhler in Python

Create a numpy array of one datetime64 per day and mark the start of the month

startdate = np.datetime64('2022-01-01T00:00:00.000000')
 # 10000 days. This array contains 0 (Day 0), 1 (Day 1), etc
day_offsets = np.arange(10000, dtype=np.int64)
usec_per_day = int(1e6) * 86400 # 86.4k sec per day, 1e6 microseconds per second
# Compute microseconds offset from first day
usec_offsets = day_offsets * usec_per_day
# Compute timestamps
timestamps = startdate + usec_offsets

# Mark start of month in boolean array
is_start_of_month = np.zeros_like(timestamps, dtype=bool)
for index, timestamp in np.ndenumerate(timestamps):
    is_start_of_month[index[0]] = timestamp.astype(datetime).day == 1

Using this method, timestamps will be the start of each day

array(['2022-01-01T00:00:00.000000', '2022-01-02T00:00:00.000000',
       '2022-01-03T00:00:00.000000', ..., '2101-12-30T00:00:00.000000',
       '2101-12-31T00:00:00.000000', '2102-01-01T00:00:00.000000'],
      dtype='datetime64[us]')

 

Posted by Uli Köhler in Data science, Python

Pandas: How to apply numpy function to every column

You can use df.transform(func, axis=0) to apply a numpy function. This leverages the fact that numpy functions work with pandas Series objects.

Example based on How to create pandas time series DataFrame example dataset:

# Load pre-built time series example dataset
df = pd.read_csv("https://datasets.techoverflow.net/timeseries-example.csv", parse_dates=["Timestamp"])
df.set_index("Timestamp", inplace=True)

# np.square will be called individually for each column
new_df = df.transform(np.square, axis=0)

Output

Original time series:

Squared time series:

Full example code

import numpy as np
import pandas as pd
from matplotlib import pyplot as plt

# Load pre-built time series example dataset
df = pd.read_csv("https://datasets.techoverflow.net/timeseries-example.csv", parse_dates=["Timestamp"])
df.set_index("Timestamp", inplace=True)

# np.sqrt will be called individually for each column
new_df = df.transform(np.square, axis=0)

# Plot subsection of original DF for better visibility
df.iloc[:len(df)//2].plot()
plt.gcf().set_size_inches(10,5)
plt.savefig("Normal-Timeseries.svg")

# Plot subsection of transformed DF for better visibility
new_df.iloc[:len(df)//2].plot()
plt.gcf().set_size_inches(10,5)
plt.savefig("Square-Timeseries.svg")

 

Posted by Uli Köhler in pandas, Python

Numpy nth root: How to

To find the nth root of number or array x in numpy use:

np.power(x, (1/n))

n can be a natural number like 4 but it can also be a floating point number like 3.26.

Example: Compute the 78th root of 1.234

>>> import numpy as np
>>> np.power(1.234, (1/78))
1.0026992894299456

 

Posted by Uli Köhler in Python

How to iterate all ISO3166-Alpha2 codes in Python

First install the iso3166 library:

pip install iso3166

Now you can iterate all the alpha2 codes like DE and US like this:

from iso3166 import countries

for alpha2 in [country.alpha2 for country in countries]:
    print(country)

This will print:

AF
AX
AL
DZ
AS
AD
AO
AI
AQ
AG
AR
AM
AW
AU
AT
AZ
BS
BH
BD
BB
BY
BE
BZ
BJ
BM
BT
BO
BQ
BA
BW
BV
BR
IO
BN
BG
BF
BI
KH
CM
CA
CV
KY
CF
TD
CL
CN
CX
CC
CO
KM
CG
CD
CK
CR
CI
HR
CU
CW
CY
CZ
DK
DJ
DM
DO
EC
EG
SV
GQ
ER
EE
ET
FK
FO
FJ
FI
FR
GF
PF
TF
GA
GM
GE
DE
GH
GI
GR
GL
GD
GP
GU
GT
GG
GN
GW
GY
HT
HM
VA
HN
HK
HU
IS
IN
ID
IR
IQ
IE
IM
IL
IT
JM
JP
JE
JO
KZ
KE
KI
KP
KR
XK
KW
KG
LA
LV
LB
LS
LR
LY
LI
LT
LU
MO
MK
MG
MW
MY
MV
ML
MT
MH
MQ
MR
MU
YT
MX
FM
MD
MC
MN
ME
MS
MA
MZ
MM
NA
NR
NP
NL
NC
NZ
NI
NE
NG
NU
NF
MP
NO
OM
PK
PW
PS
PA
PG
PY
PE
PH
PN
PL
PT
PR
QA
RE
RO
RU
RW
BL
SH
KN
LC
MF
PM
VC
WS
SM
ST
SA
SN
RS
SC
SL
SG
SX
SK
SI
SB
SO
ZA
GS
SS
ES
LK
SD
SR
SJ
SZ
SE
CH
SY
TW
TJ
TZ
TH
TL
TG
TK
TO
TT
TN
TR
TM
TC
TV
UG
UA
AE
GB
US
UM
UY
UZ
VU
VE
VN
VG
VI
WF
EH
YE
ZM
ZW

 

Posted by Uli Köhler in Python

matplotlib: How to easily format y value as percent [%]

When plotting our time series example dataset, this is the resulting plot

This post shows how to easily plot this dataset with an y axis formatted as percent. We will assume that 1.00 maps to 100%. This post is based on our previous work on Matplotlib custom SI-prefix unit tick formatter:

Note that for pandas, you need to first call df.plot() and call set_major_formatter() after that!

import matplotlib.ticker as mtick
df.plot()
plt.gca().yaxis.set_major_formatter(mtick.PercentFormatter(xmax=1.0))

If you instead want 100.0 to map to 100%, just use xmax=100.0:

import matplotlib.ticker as mtick
df.plot()
plt.gca().yaxis.set_major_formatter(mtick.PercentFormatter(xmax=1.0))

Full example

import matplotlib.ticker as mtick

# Load pre-built time series example dataset
df = pd.read_csv("https://datasets.techoverflow.net/timeseries-example.csv", parse_dates=["Timestamp"])
df.set_index("Timestamp", inplace=True)

# Plot with Y axis scaled as percent
df.plot()
plt.gca().yaxis.set_major_formatter(mtick.PercentFormatter(xmax=1.0))

 

Posted by Uli Köhler in pandas, Python

How to get first column name of pandas DataFrame

df.columns[0]

will return the name of the first column as str.

Posted by Uli Köhler in pandas, Python

How to plot multiple pandas DataFrames in a single graph

If you use code like

df1.plot()
df2.plot()

you will see both DataFrames being plotted each in their separate graphs/plots.

In order to plot both of them in a single plot, use

ax = df1.plot()
df2.plot(ax=ax)
Posted by Uli Köhler in pandas, Python

How to combine two pandas DataFrames with the same index

If you have two pandas DataFrames you want to join where the index in both DataFrames are and you want to obtain a DataFrame where the respective columns are set to NaN if there is no value from the respective DataFrame, this is typically the correct way to do it:

df_compare = pd.concat([df1, df2], axis=1, join='outer')

If you only want to keep values where both DataFrames have some value, use join='outer'

Posted by Uli Köhler in pandas, Python

How to rename a single column of a pandas DataFrame

In our previous post How to replace string in column names in Pandas DataFrame we already covered a generic method of renaming pandas columns by replacing strings.

If you just want to rename a specific column, say, the column is named value and you want to convert it to , use this snippet:

new_df = df.rename(columns=lambda s: "€" if s == "value" else s)

or use inplace=True if you want to modify the original DataFrame:

df.rename(columns=lambda s: "€" if s == "value" else s, inplace=True)

 

 

Posted by Uli Köhler in pandas, Python

How to fix pandas AttributeError: module ‘pandas’ has no attribute ‘timedelta’

Problem:

While running your Python code or Jupyter notebook, you see an error message like

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_99995/3617538487.py in <module>
      1 for date in df.index:
----> 2     print(date - pd.timedelta(5, 'years'))

/usr/local/lib/python3.9/dist-packages/pandas/__init__.py in __getattr__(name)
    259         return _SparseArray
    260 
--> 261     raise AttributeError(f"module 'pandas' has no attribute '{name}'")
    262 
    263 

AttributeError: module 'pandas' has no attribute 'timedelta'

Solution:

pandas Timedelta is spelled with a capital T: Timedelta not timedelta.

Posted by Uli Köhler in pandas, Python

How to parse date column in pandas

If you have a pandas column containing a date string

df

 

which is listed as object:

df["date"]

parse it like this:

df["date"] = pd.to_datetime(df["date"])

After which it will be listed as datetime64[ns]:

df["date"]

Posted by Uli Köhler in pandas, Python

What does sdist mean in Python?

sdist means source distribution. It is an archive (usually a .tar.gz) that contains all the source files and related data for a project in a single archive. A source distribution is what is downloaded from the PyPI package registry when you run pip install.

You can create the source distribution (sdist) for a project using

python setup.py sdist

 

Posted by Uli Köhler in Python

How to fix Python CFFI package not finding include file when running pip install

Problem:

When you install your package locally using python setup.py install, all include files are found and the package installs correctly

However you run pip install, the compiler is missing some include files

Solution:

Most likely, your include file does not end up in the sdist i.e. the source distribution .tar.gz file. In order to include it, create a MANIFEST.in file in the same folder as setup.py in order to tell setuptools what to include in addition to any file directly referenced in setup.py.

In my case, the include files are in the src folder so I add this to MANIFEST.in:

graft src/

Now, to check if that worked, run

python setup.py sdist

which will show you which files are included in the source distribution. Example:

C:\Users\ukoeh\cv_algorithms>python setup.py sdist
running sdist
running egg_info
writing cv_algorithms.egg-info\PKG-INFO
writing dependency_links to cv_algorithms.egg-info\dependency_links.txt
writing requirements to cv_algorithms.egg-info\requires.txt
writing top-level names to cv_algorithms.egg-info\top_level.txt
reading manifest file 'cv_algorithms.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'cv_algorithms.egg-info\SOURCES.txt'
running check
warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) should be supplied

creating cv_algorithms-1.0.4
creating cv_algorithms-1.0.4\cv_algorithms
creating cv_algorithms-1.0.4\cv_algorithms.egg-info
creating cv_algorithms-1.0.4\src
creating cv_algorithms-1.0.4\test
copying files to cv_algorithms-1.0.4...
copying LICENSE -> cv_algorithms-1.0.4
copying MANIFEST.in -> cv_algorithms-1.0.4
copying README.md -> cv_algorithms-1.0.4
copying setup.py -> cv_algorithms-1.0.4
copying cv_algorithms\__init__.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\_checks.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\_ffi.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\classification.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\colorspace.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\contours.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\distance.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\grassfire.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\morphology.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\neighbours.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\popcount.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\text.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\thinning.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms\utils.py -> cv_algorithms-1.0.4\cv_algorithms
copying cv_algorithms.egg-info\PKG-INFO -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\SOURCES.txt -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\dependency_links.txt -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\not-zip-safe -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\requires.txt -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying cv_algorithms.egg-info\top_level.txt -> cv_algorithms-1.0.4\cv_algorithms.egg-info
copying src\common.hpp -> cv_algorithms-1.0.4\src
copying src\distance.cpp -> cv_algorithms-1.0.4\src
copying src\grassfire.cpp -> cv_algorithms-1.0.4\src
copying src\neighbours.cpp -> cv_algorithms-1.0.4\src
copying src\popcount.cpp -> cv_algorithms-1.0.4\src
copying src\thinning.cpp -> cv_algorithms-1.0.4\src
copying src\windows.cpp -> cv_algorithms-1.0.4\src
copying test\TestColorspace.py -> cv_algorithms-1.0.4\test
copying test\TestContours.py -> cv_algorithms-1.0.4\test
copying test\TestDistance.py -> cv_algorithms-1.0.4\test
copying test\TestGrassfire.py -> cv_algorithms-1.0.4\test
copying test\TestNeighbours.py -> cv_algorithms-1.0.4\test
copying test\TestPopcount.py -> cv_algorithms-1.0.4\test
copying test\TestThinning.py -> cv_algorithms-1.0.4\test
copying test\TestUtils.py -> cv_algorithms-1.0.4\test
copying test\__init__.py -> cv_algorithms-1.0.4\test
Writing cv_algorithms-1.0.4\setup.cfg
Creating tar archive
removing 'cv_algorithms-1.0.4' (and everything under i

After that, you need to re-release your package. See my cv_algorithms project for a complete example.

Posted by Uli Köhler in Python

How to fix Python ModuleNotFoundError: No module named ‘cv2’ on Windows

Problem:

You see an error message like the following one when running some Python code on Windows:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\ukoeh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\cv_algorithms\__init__.py", line 5, in <module>
    from .text import *
  File "C:\Users\ukoeh\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\cv_algorithms\text.py", line 7, in <module>
    import cv2
ModuleNotFoundError: No module named 'cv2'

Solution:

Install the opencv-python package using

pip install opencv-python

and retry.

Posted by Uli Köhler in Python, Windows

How to fix Python CFFI error LNK2001: unresolved external symbol PyInit__…

Problem:

When trying to install your Python library using a C module on Windows, you see an error message like

LINK : error LNK2001: unresolved external symbol PyInit__cv_algorithms
build\temp.win-amd64-3.6\Release\src_cv_algorithms.cp36-win_amd64.lib : fatal error LNK1120: 1 unresolved externals
error: command 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe' failed with exit status 1120

Solution:

On Windows, CFFI does not properly generate the.

You can easily fix that by adding a template function and including that only on windows: I suggest creating windows.cpp like this:

/**
 * This contains hacks to get the installation on Windows working.
 * This fixes error LNK2001: unresolved external symbol PyInit__cv_algorithms
 */
void PyInit__cv_algorithms(void) { }

Copy & paste the name of the function from the error message! The fix will not work if you use the wrong function name.

Example how to include that file only on Windows:

platform_src = ["src/windows.cpp"] if os.name == 'nt' else []

mod_cv_algorithms = Extension('cv_algorithms._cv_algorithms',
                         sources=['src/main.cpp'] + platform_src)

For a full example, see cv_algorithms where I first implemented this fix.

Posted by Uli Köhler in Python