Programming languages

How to copy a 2D matrix to a slice of a 3D matrix in Octave

In our previous post we showed how to create a 3D matrix in Octave, for example this 2x4x3 matrix:

>> A = zeros(2, 4, 3);
>> A
A =

ans(:,:,1) =

   0   0   0   0
   0   0   0   0

ans(:,:,2) =

   0   0   0   0
   0   0   0   0

ans(:,:,3) =

   0   0   0   0
   0   0   0   0

What if we want to copy a 2D matrix slice like

>> B = ones(2, 4)
B =

   1   1   1   1
   1   1   1   1

to the 3D matrix?

First, we need to think about which matrix indices we need to copy to.
Obviously, since B is a 2x4 matrix and A is a 2x4x3 matrix, the third dimension needs to be fixed since the size of the first and second dimension of A is identical to the dimension of B.

Hence we can copy B to either

  • A(:,:,1) or
  • A(:,:,2) or
  • A(:,:,3)

For example, we can copy B to A(:,:,2) using

A(:,:,2) = B

In our example with B = ones(2,4) and A = zeros(2,4,3) will look like this:

>> A(:,:,2) = B
A =

ans(:,:,1) =

   0   0   0   0
   0   0   0   0

ans(:,:,2) =

   1   1   1   1
   1   1   1   1

ans(:,:,3) =

   0   0   0   0
   0   0   0   0

 

Posted by Uli Köhler in Octave

How to create a 3D matrix in Octave

In order to create a X*Y*Z-sized 3D matrix in Octave, use

A = zeros(X, Y, Z);

For example, to create a 5x6x3 matrix, use

A = zeros(5, 6, 3);

zeros() initializes the matrix to 0:

>> A = zeros(5, 6, 3);
>> A
A =

ans(:,:,1) =

   0   0   0   0   0   0
   0   0   0   0   0   0
   0   0   0   0   0   0
   0   0   0   0   0   0
   0   0   0   0   0   0

ans(:,:,2) =

   0   0   0   0   0   0
   0   0   0   0   0   0
   0   0   0   0   0   0
   0   0   0   0   0   0
   0   0   0   0   0   0

ans(:,:,3) =

   0   0   0   0   0   0
   0   0   0   0   0   0
   0   0   0   0   0   0
   0   0   0   0   0   0
   0   0   0   0   0   0

 

Posted by Uli Köhler in Octave

Is pypng 16-bit PNG encoding faster using pypy on the Raspberry Pi?

In our previous post How to save Raspberry Pi raw 10-bit image as 16-bit PNG using pypng we investigated how to use the pypng library to save 10-bit raw Raspberry Pi Camera images to 16-bit PNG files.

However, saving a single image took ~26 seconds using CPython 3.7.3. Since pypy can provide speedups to many Python workloads, we tried using pypy3 7.0.0 (see How to install pypy3 on the Raspberry Pi) to speed up the PNG encoding.

Results

pypng PNG export seems to be one of the workloads that are much slower using pypy3.

  • CPython 3.7.3: Encoding took 24.22 seconds
  • pypy3 7.0.0: Encoding took 266.60 seconds

Encoding is more that 10x slower when using pypy3!

Hence I don’t recommend using pypy3 to speed up pypng encoding workloads, at least not on the Raspberry Pi!

Full example

This example is derived from our full example previously posted on How to save Raspberry Pi raw 10-bit image as 16-bit PNG using pypng:

#!/usr/bin/env python3
import time
import picamera
import picamera.array
import numpy as np
import png

# Capture image
print("Capturing image...")
with picamera.PiCamera() as camera:
    with picamera.array.PiBayerArray(camera) as stream:
        camera.capture(stream, 'jpeg', bayer=True)
        # Demosaic data and write to rawimg
        # (stream.array contains the non-demosaiced data)
        rawimg = stream.demosaic()

# Write to PNG
print("Writing 16-bit PNG...")
t0 = time.time()
with open('16bit.png', 'wb') as outfile:
    writer = png.Writer(width=rawimg.shape[1], height=rawimg.shape[0], bitdepth=16, greyscale=False)
    # rawimg is a (w, h, 3) RGB uint16 array
    # but PyPNG needs a (w, h*3) array
    png_data = np.reshape(rawimg, (-1, rawimg.shape[1]*3))
    # Scale 10 bit data to 16 bit values (else it will appear black)
    # NOTE: Depending on your photo and the settings,
    #  it might still appear quite dark!
    png_data *= int(2**6)
    writer.write(outfile, png_data)
t1 = time.time()

print(f"Encoding took {(t1 - t0):.2f} seconds")

 

Posted by Uli Köhler in Python, Raspberry Pi

How to install pypy3 on the Raspberry Pi

This post shows you an easy way of getting pypy3 running on the Raspberry Pi. I used Raspbian Buster on a Raspberry Pi 3 for this example. On Raspbian buster this will install pypy3 7.x!

First install pypy3 and virtualenv:

sudo apt update && sudo apt -y install pypy3 pypy3-dev virtualenv

Now we can create a virtualenv to install pypy packages into:

virtualenv -p /usr/bin/pypy3 ~/pypy3-virtualenv

Now we can activate the virtualenv. You need to do this every time you want to use pypy, for each shell / SSH connection separately:

source ~/pypy3-virtualenv/bin/activate

If your shell prompt is now prefixed by (pypy3-virtualenv) you have successfully activated the virtualenv:

(pypy3-virtualenv) uli@raspberrypi:~ $

Now python points to pypy3 and pip will install packages locally to ~/pypy3-virtualenv.

Now you can use e.g.

python3 myscript.py

to run your script (both python and python3 will point to pypy3 if you activated the virtual environment!).

Note: Installing pypy3-dev is not strictly neccessary to get pypy3 running, but you need it in order to compile native librarie like numpy.

Posted by Uli Köhler in Python, Raspberry Pi

How to capture RaspberryPi camera 10-bit raw image in Python

You can use the picamera Python library to capture a raw sensor image of a camera attached to the Raspberry Pi via CSI:

#!/usr/bin/env python3
import picamera
import picamera.array
import numpy as np

# Capture image
print("Capturing image...")
with picamera.PiCamera() as camera:
    with picamera.array.PiBayerArray(camera) as stream:
        camera.capture(stream, 'jpeg', bayer=True)
        # Demosaic data and write to rawimg
        # (stream.array contains the non-demosaiced data)
        rawimg = stream.demosaic()

rawimg is a numpy uint16 array of dimensions (w, h, 3), e.g. (1944, 2592, 3) and contains integer values from 0 to 1023.

You can, for example, save it in a NumPy file using

np.save("rawimg.npy", rawimg) # Reload with np.load("rawimg.npy")

or save it in a compressed format using

np.savez_compressed("rawimg.npz", rawimg) # Reload with np.load("rawimg.npz")
Posted by Uli Köhler in Python, Raspberry Pi

How to fix ModuleNotFoundError: No module named ‘picamera’

Problem:

You want to run a Python script using the Raspberry Pi camera but you see an error message like

Traceback (most recent call last):
  File "mycamera.py", line 2, in <module>
    import picamera
ModuleNotFoundError: No module named 'picamera'

Solution:

You need to install the picamera Python module using pip:

sudo pip3 install picamera

or, if you are still using Python 2.x:

sudo pip install picamera

In case you see

sudo: pip3: command not found

install pip3 using

sudo apt install -y python3-pip

 

Posted by Uli Köhler in Python, Raspberry Pi

NodeJS argparse minimal example

Install argparse using

npm i --save argparse

Optional argument:

const ArgumentParser = require('argparse').ArgumentParser;
const parser = new ArgumentParser({});
parser.addArgument(['-u', '--url'], {help: "The URL to use"});
const args = parser.parseArgs();

// Example usage
console.log(args.url) // null if no such argument

If you want to make an argument mandatory, use required: true:

const ArgumentParser = require('argparse').ArgumentParser;
const parser = new ArgumentParser({});
parser.addArgument(['-u', '--url'], {help: "The URL to use", required: true});
const args = parser.parseArgs();

// Example usage
console.log(args.url)

In case the user does not provide the argument, it will print

usage: CLI.js [-h] -u URL
CLI.js: error: Argument "-u/--url" is required

Positional arguments

This positional argument will always be required:

const ArgumentParser = require('argparse').ArgumentParser;
const parser = new ArgumentParser({});
parser.addArgument(['url'], {help: "The URL to use"});
const args = parser.parseArgs();

// Example usage
console.log(args.url)

 

Posted by Uli Köhler in Javascript, NodeJS

How to just show HTTP headers using wget

Use --server-response -qO /dev/null:

wget --server-response -qO /dev/null [URL]

Example:

wget --server-response -qO /dev/null https://techoverflow.net

Example output:

HTTP/1.1 200 OK
Server: nginx/1.14.0 (Ubuntu)
Date: Mon, 18 Nov 2019 02:15:19 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: cookielawinfo-checkbox-necessary=yes; expires=Mon, 18-Nov-2019 03:15:18 GMT; Max-Age=3600; path=/
Set-Cookie: cookielawinfo-checkbox-non-necessary=yes; expires=Mon, 18-Nov-2019 03:15:18 GMT; Max-Age=3600; path=/
Link: <https://techoverflow.net/wp-json/>; rel="https://api.w.org/"

 

Posted by Uli Köhler in Shell

gzip-compress in-memory in Go

In Go, you can use the built-in gzip library together with bytes.Buffer to compress a []byte to obtain a []byte containing gzip-compressed data:

content := "Hello World!"
buf := &bytes.Buffer{}
gzWriter := gzip.NewWriter(buf)
gzWriter.Write([]byte(content))
gzWriter.Close()

This full example compresses the data in-memory and writes it to a file called text.txt.gz afterwards. Note that you can also gzip directly into a file and if you don’t need to do anything else with the compressed data, you might as well have a look at our previous post How to write to gzipped file in Go.

package main

import (
    "compress/gzip"
    "os"
)

func main() {
    content := "Hello World!"
    // Initialize gzip
    buf := &bytes.Buffer{}
    gzWriter := gzip.NewWriter(buf)
    gzWriter.Write([]byte(content))
    gzWriter.Close()
    // Convert buffer to
    ioutil.WriteFile("test.txt.gz", buf.Bytes(), 0644)
}

You can use zcat test.txt.gz to see that the content has been written to the file correctly.

Posted by Uli Köhler in Go

How to write to gzipped file in Go

This example shows you how to directly write gzip-compressed data using Go’s gzip library.

First, open the file and use gzip.NewWriter() to create a new io.Writer on it:

// Open file
f, _ := os.Create("test.txt.gz")
defer f.Close()
// Create gzip writer
gzWriter := gzip.NewWriter(f)

Now you can gzWriter.Write(). Don’t forget to gzWriter.Close()

content := "Hello World!"
gzWriter.Write([]byte(content))
gzWriter.Close()

Full example:

package main

import (
    "compress/gzip"
    "os"
)

func main() {
    // Open file
    f, _ := os.Create("test.txt.gz")
    defer f.Close()
    // Create gzip writer
    gzWriter := gzip.NewWriter(f)
    // Write content and close
    content := "Hello World!"
    gzWriter.Write([]byte(content))
    gzWriter.Close()
}

You can use zcat test.txt.gz to see that the content has been written to the file correctly.

Posted by Uli Köhler in Go

Go minimal ‘Write file’ example

This is the minimal example that writes a string (Hello World! in this example) to a file (test.txt in this example) using Go.

package main

import "io/ioutil"

func main() {
    content := "Hello World!"
    ioutil.WriteFile("test.txt", []byte(content), 0644)
}

Note that this example:

  • Creates the file with permission mode 0644, i.e. only the owner can write but others can read.
  • Encodes the text as UTF-8
Posted by Uli Köhler in Go

Go equivalent of Python’s io.BytesIO

io.BytesIO in Python 3.x provides a convenient way of having a file-like object that actually streams to/from memory.

In Go, the equivalent problem is to have a io.Reader and/or and io.Writer (i.e. the equivalent of Python’s file-like object) that is backed by a []byte or a string.

Solution: Use bytes.Buffer !

// Initialize an empty buffer (e.g. for writing)
buf := &bytes.Buffer{}

// Initialize a buffer with a []byte content (e.g. for reading)
myBytes := ...
buf := bytes.NewBuffer(myBytes)

// Initialize a buffer with string
myStr := "test 123"
buf := bytes.NewBufferString(myStr)
Posted by Uli Köhler in Go

How to download and parse HTML page in Go

This example uses goquery to request a HTML page (https://techoverflow.net) via the Go net/http client and then uses goquery and a simple CSS-style query to select the <title>...</title> HTML tag and print it’s content.

package main

import (
    "fmt"
    "log"
    "net/http"

    "github.com/PuerkitoBio/goquery"
)

func main() {
    // Perform request
    resp, err := http.Get("https://techoverflow.net")
    if err != nil {
        print(err)
        return
    }
    // Cleanup when this function ends
    defer resp.Body.Close()
    // Read & parse response data
    doc, err := goquery.NewDocumentFromReader(resp.Body)
    if err != nil {
        log.Fatal(err)
    }
    // Print content of <title></title>
    doc.Find("title").Each(func(i int, s *goquery.Selection) {
        fmt.Printf("Title of the page: %s\n", s.Text())
    })
}

Example output:

Title of the page: TechOverflow

 

Posted by Uli Köhler in Go

A simple reverse proxy example in Go

This example shows you a Go HTTP Server that forwards requests to an upstream Server using a HTTP client. Note that this example does not stream the upstream server’s responses but downloads them to RAM and then serves them to the client (hence you might need to improve upon this example if you intend to serve large files).

Also, note that this example does not implement proper error handling for the upstream request since it’s only intended to serve as a basic starting point for you to develop your own reverse proxy implementation.

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
)

func main() {
    client := &http.Client{}
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        // Perform request
        url := "https://techoverflow.net" + r.URL.Path
        req, _ := http.NewRequest(r.Method, url, r.Body)
        req.Header = r.Header
        resp, _ := client.Do(req)
        // Cleanup when this function ends
        defer resp.Body.Close()
        // Copy headers from request to response
        header := w.Header()
        for name, headers := range resp.Header {
            // Iterate all headers with one name (e.g. Content-Type)
            for _, hdr := range headers {
                header.Add(name, hdr)
            }
        }
        // Read all the response data into a []byte
        body, _ := ioutil.ReadAll(resp.Body)
        // Write header & body
        w.WriteHeader(resp.StatusCode)
        w.Write(body)
    })
    // log.Fatal shows you if there is an error like the
    //  port already being used
    fmt.Println("Listening on port 8080")
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Further reading:

Posted by Uli Köhler in Go

Iterating HTTP client response headers in Go

In this example we iterate all the HTTP headers the server sends back after a HTTP request made with the net/http Go library.

for name, headers := range resp.Header {
    // Iterate all headers with one name (e.g. Content-Type)
    for _, hdr := range headers {
        println(name + ": " + hdr)
    }
}

Full example:

package main

import (
    "net/http"
)

func main() {
    // Perform request
    resp, err := http.Get("https://ipv4.techoverflow.net/api/get-my-ip")
    if err != nil {
        print(err)
        return
    }
    // Cleanup when this function ends
    defer resp.Body.Close()
    // Read all the headers
    for name, headers := range resp.Header {
        // Iterate all headers with one name (e.g. Content-Type)
        for _, hdr := range headers {
            println(name + ": " + hdr)
        }
    }
}

This example will print, for example:

Content-Type: application/octet-stream
Content-Type: text/plain charset=UTF-8
Content-Length: 11
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, OPTIONS
Server: nginx/1.14.0 (Ubuntu)
Date: Sat, 16 Nov 2019 00:27:05 GMT

 

Posted by Uli Köhler in Go

How to get external IPv4/IPv6 address in Go

You can use TechOverflow’s IPv4 HTTP API & IPv6 HTTP API within Go:

import (
    "io/ioutil"
    "net/http"
)

func GetExternalIPv4Address() (string, error) {
    // Perform request
    resp, err := http.Get("https://ipv4.techoverflow.net/api/get-my-ip")
    if err != nil {
        return "", err
    }
    // Cleanup when this function ends
    defer resp.Body.Close()
    // Read all the response data into a []byte
    body, err := ioutil.ReadAll(resp.Body)
    // Decode & print
    return string(body), nil
}

func GetExternalIPv6Address() (string, error) {
    // Perform request
    resp, err := http.Get("https://ipv6.techoverflow.net/api/get-my-ip")
    if err != nil {
        return "", err
    }
    // Cleanup when this function ends
    defer resp.Body.Close()
    // Read all the response data into a []byte
    body, err := ioutil.ReadAll(resp.Body)
    // Decode & print
    return string(body), nil
}

Usage example:

package main

func main() {
    v4, _ := GetExternalIPv4Address()
    v6, _ := GetExternalIPv6Address()
    println(v4)
    println(v6)
}

If you save both files in a directory named GoIPAddress, you can run

$ go build
$ ./GOIPAddress
31.190.168.110
2a03:4012:2:1022::1

 

 

Posted by Uli Köhler in Go

Go HTTP client minimal example

This example gets the current IPv4 address from TechOverflow’s IP address HTTP API How to get your current IPv4 address using wget.

package main

import (
    "io/ioutil"
    "net/http"
)

func main() {
    // Perform request
    resp, err := http.Get("https://ipv4.techoverflow.net/api/get-my-ip")
    if err != nil {
        print(err)
        return
    }
    // Cleanup when this function ends
    defer resp.Body.Close()
    // Read all the response data into a []byte
    body, err := ioutil.ReadAll(resp.Body)
    // Decode & print
    println(string(body))
}

Save as Main.go in a new project directory (e.g. GoHttpTest) and run go build in that directory.
After that, you can run ./GoHttpTest which should print your IPv4 adress. Example:

uli@server ~/GoHttpTest % go build
uli@server ~/GoHttpTest % ./GoHttpTest 
91.59.80.56

 

Posted by Uli Köhler in Allgemein, Go

Simulating survival data for Kaplan-Meier plots in Python

Libraries like lifelines provide a plethora of example datasets that one can work with. However, for many tasks you need to simulate specific behaviour in survival curves.

In this post, we demonstrate a simple algorithm to generate survival data in a format comparable to the one used in the lifelines example datasets like load_leukemia().

The generation algorithm is based on the following assumptions:

  • There is a strict survival plateau with a given survival probability starting at a given point in time
  • The progression from 100% survival, t=0 to the survival plateau is approximately linear (i.e. if you would generate an infinite number of datapoints, the survival curve would be linear)
  • No censoring events shall be generated except for censoring all surviving participants at the end point of the timeline.

Code:

import numpy as np
import random
from lifelines import KaplanMeierFitter

def simulate_survival_data_linear(N, survival_plateau, t_plateau, t_end):
    """
    Generate random simulated survival data using a linear model
    
    Keyword parameters
    ------------------
    N : integer
        Number of entries to generate
    survival_plateau : float
        The survival probability of the survival plateau
    t_plateau : float
        The time point where the survival plateau starts
    t_end : float
        The time point where all surviving participants will be censored.
    
    Returns
    -------
    A dict with "Time" and "Event" numpy arrays: 0 is censored, 1 is event
    """
    data = {"Time": np.zeros(N), "Event": np.zeros(N)}

    for i in range(N):
        r = random.random()
        if r <= survival_plateau:
            # Event is censoring at the end of the time period
            data["Time"][i] = t_end
            data["Event"][i] = 0
        else: # Event occurs
            # Normalize where we are between 100% and the survival plateau
            p = (r - survival_plateau) / (1 - survival_plateau)
            # Linear model: Time of event linearly depends on uniformly & randomly chosen position
            #  in range (0...tplateau)
            t = p * t_plateau
            data["Time"][i] = t
            data["Event"][i] = 1
    return data

# Example usage
data1 = simulate_survival_data_linear(250, 0.2, 18, 24)
data2 = simulate_survival_data_linear(250, 0.4, 17.2, 24)

Given data1 and data2 (see the usage example at the end of the code) you can plot them using

# Plot bad subgroup
kmf1 = KaplanMeierFitter()
kmf1.fit(data1["Time"], event_observed=data1["Event"], label="Bad subgroup")
ax = kmf1.plot()

# Plot good subgroup
kmf2 = KaplanMeierFitter()
kmf2.fit(data2["Time"], event_observed=data2["Event"], label="Good subgroup")
ax = kmf2.plot(ax=ax)

# Set Y axis to fixed scale
ax.set_ylim([0.0, 1.0])

Thi

Do not want a survival plateau?

Just set t_end = t_survival:

# Example usage
data1 = simulate_survival_data_linear(250, 0.2, 24, 24)
data2 = simulate_survival_data_linear(250, 0.4, 24, 24)
# Code to plot: See above

What happens if you have a low number of participants?

Let’s use 25 instead of 250 as above:

# Example usage
data1 = simulate_survival_data_linear(25, 0.2, 24, 24)
data2 = simulate_survival_data_linear(25, 0.4, 24, 24)
# Plot code: See above

Although we generated the data with the same data, the difference is much less clear in this example, especially towards the end of the timeline (note however that the data is generated randomly, so you might see a different result). You can see a large portion of the confidence intervals overlappings near t=24. In other words, based on this data it is not clear that the two groups of patients are significantly different (in other words, P \geq 0.05)

Posted by Uli Köhler in Python, Statistics

How to use custom legend label in lifelines Kaplan-Meier plot?

Using the lifelines library, you can easily plot Kaplan-Meier plots, e.g. as seen in our previous post Minimal Python Kaplan-Meier Plot example:

from lifelines.datasets import load_leukemia
from lifelines import KaplanMeierFitter
df = load_leukemia()

kmf = KaplanMeierFitter()
kmf.fit(df['t'], df['Rx']) # t = Timepoints, Rx: 0=censored, 1=event
kmf.plot()

What if you want a custom label instead of KM_estimates to appear in the legend?

Use kmf.fit(..., label='Your label'). Since we use the leukemias dataset for this example, we use the label 'Leukemia'

Full example:

from lifelines.datasets import load_leukemia
from lifelines import KaplanMeierFitter
# Load datasets
df_leukemia = load_leukemia()

# Fit & plot leukemia dataset
kmf_leukemia = KaplanMeierFitter()
kmf_leukemia.fit(df_leukemia['t'], df_leukemia['Rx'], label="Leukemia")
ax = kmf_leukemia.plot()
# Set Y axis to fixed scale
ax.set_ylim([0.0, 1.0])

Posted by Uli Köhler in Python, Statistics

How plot multiple Kaplan-Meier curves using lifelines

Using the lifelines library, you can easily plot Kaplan-Meier plots, e.g. as seen in our previous post Minimal Python Kaplan-Meier Plot example:

from lifelines.datasets import load_leukemia
from lifelines import KaplanMeierFitter
df = load_leukemia()

kmf = KaplanMeierFitter()
kmf.fit(df['t'], df['Rx']) # t = Timepoints, Rx: 0=censored, 1=event
kmf.plot()

What if you want to plot multiple survival curves?

The call to kmf.plot() returns a Matplotlib ax object which you can use on a second kmf2.plot() call als argument: kmf2.plot(ax=ax).

Full example (note that we also set a fixed Y range of [0.0, 1.0], see How to make Y axis start from 0 in lifelines Kaplan-Meier plots):

from lifelines.datasets import load_leukemia, load_lymphoma
from lifelines import KaplanMeierFitter
# Load datasets
df_leukemia = load_leukemia()
df_lymphoma = load_lymphoma()

# Fit & plot leukemia dataset
kmf_leukemia = KaplanMeierFitter()
kmf_leukemia.fit(df_leukemia['t'], df_leukemia['Rx'], label="Leukemia")
ax = kmf_leukemia.plot()

# Fit & plot lymphoma dataset
kmf_lymphoma = KaplanMeierFitter()
kmf_lymphoma.fit(df_lymphoma['Time'], df_lymphoma['Censor'], label="Lymphoma")
ax = kmf_lymphoma.plot(ax=ax)

# Set Y axis to fixed scale
ax.set_ylim([0.0, 1.0])

Posted by Uli Köhler in Python, Statistics