How to Convert a MariaDB SQL Dump to Parquet Using a Docker Container

This post shows how to turn a .sql dump (mysqldump/MariaDB backup) into a .parquet file without installing a database server — by spinning up a throwaway MariaDB container, importing, querying, and exporting with pandas.

Procedure

1. Start a temporary MariaDB container

start-db.sh

docker run -d --rm --name tmp-mariadb \
  -e MARIADB_ROOT_PASSWORD=root \
  -p 33306:3306 \
  mariadb:11

# Wait for it to accept connections
until docker exec tmp-mariadb mariadb -uroot -proot -e "SELECT 1" >/dev/null 2>&1; do
  sleep 1
done

2. Import the dump

import.sh

docker exec -i tmp-mariadb mariadb -uroot -proot -e "CREATE DATABASE mydb"
docker exec -i tmp-mariadb mariadb -uroot -proot mydb < backup.sql

For gzipped dumps: zcat backup.sql.gz | docker exec -i tmp-mariadb mariadb -uroot -proot mydb.

3. Query into Parquet with pandas

dump_to_parquet.py

import pandas as pd
import sqlalchemy

engine = sqlalchemy.create_engine(
    "mysql+pymysql://root:root@127.0.0.1:33306/mydb"
)

df = pd.read_sql("""
    SELECT SerialNo, CANAddress, Name, ZoneName, Type,
           SetVoltage, SetCurrent
    FROM nodes
    ORDER BY SerialNo
""", engine)

df.to_parquet("nodes.parquet", index=False)
print(f"Wrote {len(df)} rows")

Dependencies: pip install pandas sqlalchemy pymysql pyarrow.

4. Tear down

cleanup.sh

docker stop tmp-mariadb   # --rm removes it automatically

Whole pipeline in one script

sql_to_parquet.py

#!/usr/bin/env python3
"""Convert a MariaDB .sql dump to .parquet via a throwaway container."""
import subprocess, sys, time
import pandas as pd
import sqlalchemy

DUMP, OUT, QUERY = sys.argv[1], sys.argv[2], sys.argv[3]
NAME = "tmp-mariadb-convert"

subprocess.run(["docker", "run", "-d", "--rm", "--name", NAME,
                "-e", "MARIADB_ROOT_PASSWORD=root", "-p", "33306:3306",
                "mariadb:11"], check=True)
try:
    # wait for ready
    for _ in range(60):
        r = subprocess.run(["docker", "exec", NAME, "mariadb",
                            "-uroot", "-proot", "-e", "SELECT 1"],
                           capture_output=True)
        if r.returncode == 0:
            break
        time.sleep(1)
    subprocess.run(["docker", "exec", "-i", NAME, "mariadb",
                    "-uroot", "-proot", "-e", "CREATE DATABASE mydb"], check=True)
    with open(DUMP, "rb") as f:
        subprocess.run(["docker", "exec", "-i", NAME, "mariadb",
                        "-uroot", "-proot", "mydb"], stdin=f, check=True)
    engine = sqlalchemy.create_engine(
        "mysql+pymysql://root:root@127.0.0.1:33306/mydb")
    df = pd.read_sql(QUERY, engine)
    df.to_parquet(OUT, index=False)
    print(f"{OUT}: {len(df)} rows")
finally:
    subprocess.run(["docker", "stop", NAME], capture_output=True)

Usage:

usage.sh

python3 sql_to_parquet.py backup.sql nodes.parquet "SELECT * FROM nodes"

Notes

--rm + explicit docker stop means zero cleanup even on failure.
If the dump is large, raise the wait loop and consider --max-allowed-packet / innodb_buffer_pool_size flags on the container.
Multiple dumps → run the script in a loop; each run is independent.
Works identically for MySQL dumps — swap the image to mysql:8 and the client commands accordingly.