CoreOS

Step 1: Install pip

sudo rpm-ostree install python3-pip

then reboot for the changes to take effect:

sudo systemctl reboot

Step 2: Install magic-wormhole

sudo pip install magic-wormhole

In CoreOS, run

sudo rpm-ostree status

and look for the entry with the dot (●) in front of it to see which deployment – i.e. which CoreOS version is currently active. Then, look for Version:  in the line below. This serves as the alternative to lsb_release -a which is not available on CoreOS.

Example output:

State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Thu 2022-12-08 03:49:05 UTC)
Deployments:
● fedora:fedora/x86_64/coreos/stable
                  Version: 37.20221106.3.0 (2022-11-28T20:05:48Z)
               BaseCommit: 6278bd1e5f311880a6975307e7ce734076a0b1a37f8a97c875c07037c748ddcc
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
          LayeredPackages: bmon docker-compose htop iotop make tailscale tree wget xe-guest-utilities-latest

  fedora:fedora/x86_64/coreos/stable
                  Version: 36.20221030.3.0 (2022-11-11T15:51:02Z)
               BaseCommit: eab21e5b533407b67b1751ba64d83c809d076edffa1ff002334603bf13655a14
             GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
          LayeredPackages: bmon docker-compose htop iotop make tailscale tree wget xe-guest-utilities-latest

In this example, CoreOS 37.20221106.3.0 is active.

When running

rpm-ostree ex apply-live

you will see the following error message in newer versions of CoreOS:

error: Unknown "ex" subcommand "apply-live"

The new equivalent of rpm-ostree ex apply-live is

rpm-ostree ex livefs --i-like-danger

Problem:

When trying to install a package using rpm-ostree, you see an error message like

error: Transaction in progress: deploy --lock-finalization revision=5040eaabed46962a07b1e918ba5afa1502e1f898bf958673519cd83e986c228f --disallow-downgrade 

Solution:

The error message means that currently there’s an rpm-ostree operating in progress and you need to wait for it to finish.

In order to see which process is running, use

ps aux | grep rpm

Example output:

[root@CoreOS uli]# ps aux | grep rpm
root         730 41.2  1.7 1218036 34568 ?       Ssl  18:41   0:30 /usr/bin/rpm-ostree start-daemon
zincati     1896  0.0  0.8 481172 17324 ?        Sl   18:41   0:00 rpm-ostree deploy --lock-finalization revision=5040eaabed46962a07b1e918ba5afa1502e1f898bf958673519cd83e986c228f --disallow-downgrade
root        3223  0.0  0.0 221452   832 pts/0    S+   18:42   0:00 grep --color=auto rpm

As you can see in the second line:

zincati 1896 0.0 0.8 481172 17324 ? Sl 18:41 0:00 rpm-ostree deploy --lock-finalization revision=5040eaabed46962a07b1e918ba5afa1502e1f898bf958673519cd83e986c228f --disallow-downgrade

the user zincati is currently running rpm-ostree on my system. zincati is the Fedora CoreOS auto-updater – in other words, an automatic system update is currently running on CoreOS.

In case the process got stuck and waiting doesn’t help reboot the system. Killing the process won’t work.

Problem:

When logging into your CoreOS instance, you see this warning message:

############################################################################
WARNING: This system is using cgroups v1. For increased reliability
it is strongly recommended to migrate this system and your workloads
to use cgroups v2. For instructions on how to adjust kernel arguments
to use cgroups v2, see:
https://docs.fedoraproject.org/en-US/fedora-coreos/kernel-args/

To disable this warning, use:
sudo systemctl disable coreos-check-cgroups.service
############################################################################

but when you look at https://docs.fedoraproject.org/en-US/fedora-coreos/kernel-args/ you only see an example of how to initialize a new CoreOS instance with Ignition files with cgroups v2.

Solution:

In order to migrate your system to cgroups v2, run

sudo rpm-ostree kargs --delete=systemd.unified_cgroup_hierarchy

After that, you need to reboot your system in order for the changes to take effect:

sudo systemctl reboot

After the system has rebooted, the error should disappear.

In constrast to other Linux-based systems, CoreOS requires quite a large learning curve to get installed properly – for example, you have to create the right ignition file for . This is a huge obstacle to overcome especially for first-time users.

This posts attempts to alleviate the steep learning curve by providing a basic config that is suitable for most practical (and especially small-scale) usecases and provides a good starting point for custom configs.

Simple install

First, boot up the VM from the CoreOS Live CD. We assume that you have a DHCP network connected to eth0. You will see a shell immediately.

The VM will automatically acquire an IP address over DHCP.

You can use TechOverflow’s hosted ignition file for the installation. You need to use the correct disk instead of /dev/xvda depending on your hardware/hypervisor. If in doubt, use lsblk to find the correct disk name.

Now run the installation command:

sudo coreos-installer install /dev/xvda --copy-network --ignition-url https://techoverflow.net/coreos.ign

After the installation is finished, reboot using

reboot

and the machine has rebooted, you can use the default login credentials:

Username: admin
Password: coreos

The hostname is CoreOS.

You absolutely need to change the password after the installation! If you create another user, remember that you still need to change the password of the admin user using

sudo passwd admin

Build your own config file

This is the Ignition YAML we used to create the correct config file. Use our online transpiler at https://fcct.techoverflow.net to compile the YAML to the JSON file. In order to create a new password hash, use TechOverflow’s docker-based mkpasswd approach.

variant: fcos
version: 1.0.0
passwd:
  users:
    - name: admin
      groups:
        - "sudo"
        - "docker"
      password_hash: $y$j9T$n6h8P2ik8tfoNUFBBoly00$7bnrMF8oFrB25Fc3NqigqEH/MI5YXIJwtCG/iEsns.2

systemd:
  units:
    - name: docker.service
      enabled: true

    - name: containerd.service
      enabled: true
    - name: [email protected]
      dropins:
      - name: autologin-core.conf
        contents: |
          [Service]
          # Override Execstart in main unit
          ExecStart=
          # Add new Execstart with `-` prefix to ignore failure
          ExecStart=-/usr/sbin/agetty --autologin admin --noclear %I $TERM
          TTYVTDisallocate=no
storage:
  files:
    - path: /etc/hostname
      mode: 0644
      contents:
        inline: |
          CoreOS
    - path: /etc/profile.d/systemd-pager.sh
      mode: 0644
      contents:
        inline: |
          # Tell systemd to not use a pager when printing information
          export SYSTEMD_PAGER=cat
    - path: /etc/sysctl.d/20-silence-audit.conf
      mode: 0644
      contents:
        inline: |
          # Raise console message logging level from DEBUG (7) to WARNING (4)
          # to hide audit messages from the interactive console
          kernel.printk=4
    - path: /etc/ssh/sshd_config.d/20-enable-passwords.conf
      mode: 0644
      contents:
        inline: |
          # Enable SSH password login
          PasswordAuthentication yes

which results in the following transpiled JSON:

{
  "ignition": {
    "version": "3.0.0"
  },
  "passwd": {
    "users": [
      {
        "groups": [
          "sudo",
          "docker"
        ],
        "name": "admin",
        "passwordHash": "$y$j9T$n6h8P2ik8tfoNUFBBoly00$7bnrMF8oFrB25Fc3NqigqEH/MI5YXIJwtCG/iEsns.2"
      }
    ]
  },
  "storage": {
    "files": [
      {
        "contents": {
          "source": "data:,CoreOS%0A"
        },
        "mode": 420,
        "path": "/etc/hostname"
      },
      {
        "contents": {
          "source": "data:,%23%20Tell%20systemd%20to%20not%20use%20a%20pager%20when%20printing%20information%0Aexport%20SYSTEMD_PAGER%3Dcat%0A"
        },
        "mode": 420,
        "path": "/etc/profile.d/systemd-pager.sh"
      },
      {
        "contents": {
          "source": "data:,%23%20Raise%20console%20message%20logging%20level%20from%20DEBUG%20(7)%20to%20WARNING%20(4)%0A%23%20to%20hide%20audit%20messages%20from%20the%20interactive%20console%0Akernel.printk%3D4%0A"
        },
        "mode": 420,
        "path": "/etc/sysctl.d/20-silence-audit.conf"
      },
      {
        "contents": {
          "source": "data:,%23%20Enable%20SSH%20password%20login%0APasswordAuthentication%20yes%0A"
        },
        "mode": 420,
        "path": "/etc/ssh/sshd_config.d/20-enable-passwords.conf"
      }
    ]
  },
  "systemd": {
    "units": [
      {
        "enabled": true,
        "name": "docker.service"
      },
      {
        "enabled": true,
        "name": "containerd.service"
      },
      {
        "dropins": [
          {
            "contents": "[Service]\n# Override Execstart in main unit\nExecStart=\n# Add new Execstart with `-` prefix to ignore failure\nExecStart=-/usr/sbin/agetty --autologin admin --noclear %I $TERM\nTTYVTDisallocate=no\n",
            "name": "autologin-core.conf"
          }
        ],
        "name": "[email protected]"
      }
    ]
  }
}

Do you want to install Fedora CoreOS packages without having to reboot your entire system in order for the packages to be available? Just run

sudo rpm-ostree ex apply-live

after running your rpm-ostree install commands.

For example:

sudo rpm-ostree install nano
sudo rpm-ostree ex apply-live

Note that this is not completely safe for multiple reasons, not even for seemingly innocuous utility packages like nano:

• As indicated by the ex in the command, the apply-live command is experimental
• It might apply other changes from the new OSTree like automatically installed updated and hence might have effects
• When changing files on a system with productive services running, the services might crash or experience other issues. This might not happen immediately and it might be hard to debug especially in a complex environment. In case you want to safely update your services, it’s almost always best to just reboot into the new OSTree.

Also read our previous post on Why do you have to reboot after rpm-ostree install on Fedora CoreOS? where we explain the technical reasoning behind the reboots.

If you have worked with Fedora CoreOS, you might have noticed that every time you install a package you need to reboot in order for the files from said package to be available to you. This is quite different from other Linux distributions where you can immediately use whatever package you installed without having to reboot every time.

What is the technical reasoning for having to reboot?

rpm-ostree is quite a special tool: It does not just install a package. This has the advantage that the currently running system is not modified at all, but a separate OS tree – image it like an image containing all the files constituting your system – is built after running rpm-ostree install.

While rebooting after every install might seem like a stupid idea since it takes down the entire server, remember that it can save you a lot of headache since there are no partially updated services and you don’t need to manually fix or restart anything since everything is restarted on reboot. This means that your system is always in a consistent state, since every service is cleanly shut down before the system reboot – and after the reboot, every service is cleanly started with the system changes.

Can you install multiple packages before having to reboot?

Yes, you can run multiple rpm-ostree install commands before rebooting. When rebooting, all the changes will be applied at once.

Can you delay the reboot after rpm-ostree install?

Yes, there is no need to reboot immediately after the rpm-ostree command. You can delay the reboot as long as you like. Note however, that when the machine is rebooted for reasons other than a manual reboot (like a power outage or restart of the VM host), the updates will be applied as well, but you might not be there to check if all services are running correctly. Hence, I recommend to reboot as soon as possible.

Can you avoid to reboot after installing packages?

Yes, Fedora CoreOS provides an experimental live update feature using rpm-ostree ex apply-live. See our post How to apply Fedora CoreOS changes without a reboot . Note that applying updates or new packages on a system with productively running services might be a bad idea, but it’s not inherently more unsafe than installing packages on a typical Linux distribution like Debian, Fedora or Ubuntu where every install or update to a package immediately affects the files on the file system.

Just install it using rpm-ostree:

sudo rpm-ostree install docker-compose

and then reboot in order for the changes to the OSTree to take effect:

sudo systemctl reboot

In Fedora CoreOS, you can install the Xen guest utilities using

sudo rpm-ostree install xe-guest-utilities-latest

After installing the package, reboot in order for the changes to take effect:

sudo systemctl reboot

Now we need to enable and start the Xen service:

sudo systemctl enable --now xe-linux-distribution

It will now automatically start on boot.

Example output from the install command:

# rpm-ostree install xe-guest-utilities-latest
Checking out tree 49ec34c... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-08-25T19:10:34Z
rpm-md repo 'updates' (cached); generated: 2021-05-13T01:04:01Z
rpm-md repo 'fedora' (cached); generated: 2020-10-19T23:27:19Z
Importing rpm-md... done
Resolving dependencies... done
Will download: 1 package (1.0 MB)
Downloading from 'updates'... done
Importing packages... done
Checking out packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Added:
  xe-guest-utilities-latest-7.21.0-1.fc33.x86_64
Run "systemctl reboot" to start a reboot

If you want to use the German keyboard layout in the Fedora CoreOS installer, set the de keymap using:

sudo localectl set-keymap de

They new keymap will be effective immediately.

Note that the keyboard layout will not automatically be transferred to the installed system.

In order to set the keymap in the Fedora CoreOS installation shell, use

sudo localectl set-keymap [keymap]

For example, in order to set the de keymap:

sudo localectl set-keymap de

Add this config line to your Fedora CoreOS Ignition config in order to enable SSH password authentication on your install.

storage:
  files:
    - path: /etc/ssh/sshd_config.d/20-enable-passwords.conf
      mode: 0644
      contents:
        inline: |
          PasswordAuthentication yes

By default, Fedora CoreOS will only allow pubkey authentication and disable password authentication. This Ignition config will set PasswordAuthentication yes as a config option for the SSH daemon.

Original source: The Fedore CoreOS authentication guide.