pandas

TL;DR: Use our pre-built example dataset like this:

# Load pre-built time series example dataset
df = pd.read_csv("https://datasets.techoverflow.net/timeseries-example.csv", parse_dates=["Timestamp"])
df.set_index("Timestamp", inplace=True)

How to build your own time series example dataset

In our previous post Easily generate sine/cosine waveform data in Python using UliEngineering we showed how to generate sine and cosine waves using UliEngineering.

In this post, we show how to create a pandas DataFrame containing sine and cosine data to be used as a sample time series dataset.

First, we generate the sine and cosine wave data:

import pandas as pd
import numpy as np
from UliEngineering.SignalProcessing.Simulation import sine_wave, cosine_wave

# Configure the properties of the sine wave here
frequency = 10.0 # 10 Hz sine / cosine wave
samplerate = 10000 # 10 kHz
nseconds = 1 # Generate 1 second of data

sine = sine_wave(frequency=frequency, samplerate=samplerate, length=nseconds)
cosine = cosine_wave(frequency=frequency, samplerate=samplerate, length=nseconds)
nsamples = len(sine) # How many values we have in the data arrays

After that, we define the timestamp where the dataset starts:

start_timestamp = pd.Timestamp('now')

Now we can create a list of Timestamp objects representing the points in time where the signal has been sampled:

# Create timestamps by offsetting
timedelta = pd.Timedelta(1/samplerate, 'seconds')
timestamps =  [start_timestamp + i * timedelta for i in range(nsamples)]

Now we’re reading to create the DataFrame object:

df = pd.DataFrame(index=timestamps, data={
    "Sine": sine,
    "Cosine": cosine
})
df.index.name = 'Timestamp'

Now we can use df.plot() to plot the dataset:

# Use nice plotting style
from matplotlib import pyplot as plt
plt.style.use("ggplot")
# Plot dataset
df.plot()
# Make figure larger
plt.gcf().set_size_inches(10, 5)

Additionally we can export the dataset as CSV using

df.to_csv("/ram/timeseries-example.csv")

This example file is also available online at https://techoverflow.net/datasets/timeseries-example.csv

Full example:

#!/usr/bin/env python3
import pandas as pd
import numpy as np
from UliEngineering.SignalProcessing.Simulation import sine_wave, cosine_wave
# Configure the properties of the sine wave here
frequency = 10.0 # 10 Hz sine / cosine wave
samplerate = 10000 # 10 kHz
nseconds = 1 # Generate 1 second of data
sine = sine_wave(frequency=frequency, samplerate=samplerate, length=nseconds)
cosine = cosine_wave(frequency=frequency, samplerate=samplerate, length=nseconds)
nsamples = len(sine) # How many values we have in the data arrays

start_timestamp = pd.Timestamp('now')

# Create timestamps by offsetting
timedelta = pd.Timedelta(1/samplerate, 'seconds')
timestamps =  [start_timestamp + i * timedelta for i in range(nsamples)]

df = pd.DataFrame(index=timestamps, data={
    "Sine": sine,
    "Cosine": cosine
})
df.index.name = 'Timestamp'

df.to_csv("timeseries-example.csv")

In our previous post we showed how to subtract 5 minutes from a pandas DataFrame:

pd.Timestamp('now') - pd.Timedelta(10, 'minutes')

We can also use this knowledge in order to get the last 10 minutes of a pandas DataFrame. In our example, we assume that df[“Timestamp”] contains the timestamp. First, we get the last timestamp in the dataset using

# Use this if the timestamp is the index of the DataFrame
last_ts = df.index.iloc[-1]

or

# ... or use this if the timestamp is in a colum
last_ts = df["Timestamp"].iloc[-1]

Next, we define the first timestamp that shall be considered by subtracting 10 minutes from last_ts:

first_ts = last_ts - pd.Timedelta(10, 'minutes')

Now we can filter the DataFrame using

# Use this if the Timestamp is in a column
filtered_df = df[df["Timestamp"] >= first_ts]

or

# Use this if the Timestamp is the index of the DataFrame
filtered_df = df[df.index >= first_ts]

By filtering, we don’t need the DataFrame to be sorted and the original order will be maintained.

Full example:

This example loads our pre-built time series example dataset from our previous post How to create pandas time series DataFrame example dataset. The code loads that dataset (which is 1 second long) and takes the last 0.5 seconds from it.

import pandas as pd

# Load example dataset
df = pd.read_csv("https://techoverflow.net/datasets/timeseries-example.csv", parse_dates=["Timestamp"])
df.set_index("Timestamp", inplace=True)

# Use this if the timestamp is the index of the DataFrame
last_ts = df.index[-1]

first_ts = last_ts - pd.Timedelta(0.5, 'seconds')

filtered_df = df[df.index >= first_ts]

# Plot the result
filtered_df.plot()

In our previous post we showed how to create a pandas Timestamp representing the current point in time:

pd.Timestamp('now')

You can subtract 5 minutes from that timestamp by using Timedelta(5, 'minutes'):

pd.Timestamp('now') - pd.Timedelta(5, 'minutes')

In order to create a pandas Timestamp representing the current point in time, use

pd.Timestamp('now')

This will create a Timestamp in the current timezone.

Full example:

import pandas as pd

now = pd.Timestamp('now')

print(now) # Prints e.g. Timestamp('2020-05-25 19:02:31.051836')

Use .iloc[-1] to get the last row (all columns) of a pandas DataFrame, for example:

my_dataframe.iloc[-1]

Use .iloc[-1] to get the last element of a pandas Series, for example:

my_dataframe['MyColumn'].iloc[-1]

If you have a pandas.Timedelta object, you can use Timedelta.total_seconds() to get the seconds as a floating-point number with millisecond resolution and then multiply with one billion (1e3, the number of milliseconds in one second) to obtain the number of milliseconds in the Timedelta:

timedelta.total_seconds() * 1e3

In case you want an integer, use

int(round(timedelta.total_seconds() * 1e3))

Note that using round() is required here to avoid errors due to floating point precision.

or use this function definition:

def milliseconds_from_timedelta(timedelta):
    """Compute the milliseconds in a timedelta as floating-point number"""
    return timedelta.total_seconds() * 1e3

def milliseconds_from_timedelta_integer(timedelta):
    """Compute the milliseconds in a timedelta as integer number"""
    return int(round(timedelta.total_seconds() * 1e3))

# Usage example:
ms = milliseconds_from_timedelta(timedelta)
print(ms) # Prints 2000.752
ms = milliseconds_from_timedelta_integer(timedelta)
print(ms) # Prints 2001

If you have a pandas.Timedelta object, you can use Timedelta.total_seconds() to get the seconds as a floating-point number with microsecond resolution and then multiply with one million (1e6, the number of microseconds in one second) to obtain the number of microseconds in the Timedelta:

timedelta.total_seconds() * 1e6

In case you want an integer, use

int(round(timedelta.total_seconds() * 1e6))

Note that using round() is required here to avoid errors due to floating point precision.

or use this function definition:

def microseconds_from_timedelta(timedelta):
    """Compute the microseconds in a timedelta as floating-point number"""
    return timedelta.total_seconds() * 1e6

def microseconds_from_timedelta_integer(timedelta):
    """Compute the microseconds in a timedelta as integer number"""
    return int(round(timedelta.total_seconds() * 1e6))

# Usage example:
us = microseconds_from_timedelta(timedelta)
print(us) # Prints 2000751.9999999998

us = microseconds_from_timedelta_integer(timedelta)
print(us) # Prints 2000752

If you have a pandas.Timedelta object, you can use Timedelta.total_seconds() to get the seconds as a floating-point number with nanosecond resolution and then multiply with one billion (1e9, the number of nanoseconds in one second) to obtain the number of nanoseconds in the Timedelta:

timedelta.total_seconds() * 1e9

In case you want an integer, use

int(round(timedelta.total_seconds() * 1e9))

Note that using round() is required here to avoid errors due to floating point precision.

or use this function definition:

def nanoseconds_from_timedelta(timedelta):
    """Compute the nanoseconds in a timedelta as floating-point number"""
    return timedelta.total_seconds() * 1e9

def nanoseconds_from_timedelta_integer(timedelta):
    """Compute the nanoseconds in a timedelta as integer number"""
    return int(round(timedelta.total_seconds() * 1e9))

# Usage example:
ns = nanoseconds_from_timedelta(timedelta)
print(ns) # Prints 2000751999.9999998

ns = nanoseconds_from_timedelta_integer(timedelta)
print(ns) # Prints 2000752000

If you have two pandas.Timestamp objects, you can simply substract them using the minus operator (-) in order to obtain a pandas.Timedelta object:

import pandas as pd
import time

# Create two timestamps
ts1 = pd.Timestamp.now()
time.sleep(2)
ts2 = pd.Timestamp.now()

# The difference of these timestamps is a pandas.Timedelta object.
timedelta = ts2 - ts1
print(timedelta) # Prints '0 days 00:00:02.000752'

Use

import pandas as pd

current_timestamp = pd.Timestamp.now()

pd.Timestamp.now() will return the timestamp as pandas.Timestamp object.

Example:

>>> import pandas as pd
>>> pd.Timestamp.now()
Timestamp('2019-10-27 16:11:43.998993')

Problem:

You need to convert pandas DataFrame to a LaTeX table. You want to include the pandas column and row labels in the table.

Continue reading →