Linux

Apr

2019

What does ‘sudo usermod -a -G group $USER’ do on Linux?

In our posts, especially posts like Solving Docker permission denied while trying to connect to the Docker daemon socket you can often see commands like

sudo usermod -a -G docker $USER

But what does this command actually do on your system?

Let’s break it down:

sudo means: Run this command as root. This is required for usermod since usually only root can modify which groups a user belongs to
usermod is a command that modifies the system configuration for a specific user ($USER in our example – see below). See the manpage documentation for more details on what you can do with it!
-a is a shortcut for --append: It means append the group to the list of groups the user belongs to!
-G is a shortcut for --groups: It tells usermod that the next argument is a group. Note that you need to use a capital -G here because we don’t want to modify the user’s primary group but the list of supplemental groups the user belongs to. See the Primary and supplemental groups section below for more details.
docker is the group we want to add $USER to. This could be any Linux group, provided that it exists. Use less /etc/group to have a look at all the groups that exist!
$USER is the user that we want to modify. $USER is a shell shortcut for the user that is running the command. This works even when using sudo (i.e. if your user is named uli and you are running sudo usermod -a -G docker $USER, the user uli will be added to the docker group, not the user root even though the command is run as root). You can also use a specific username instead of $USER, e.g. sudo usermod -a -G docker john to add the user john to the docker group

Primary and supplemental groups

When you browse through the usermod manpage, you’ll see there’s -G which adds a group to a user’s list of supplementary groups, and there’s -g which modifies a user’s primary group.

The pragmatic answer is: If you need to ask, you’ll always need to use -G.

Having to modify the primary group of a user is extremely rare in my experience. The purpose of primary groups existing is mainly that if you create a file, Linux needs to know which group it belongs to by default (i.e. if you don’t explicitly specify a group).

See this AskUbuntu post for more details on the purpose of primary and supplemental groups.

Apr

2019

Simple systemd timer generator

Use this generator to generate a systemd service and a corresponding timer:

Continue reading →

Apr

2019

Are changes made to mmap MAP_PRIVATE visible to the current process?

The mmap mapage tells us that whether changes in memory made to a MAP_PRIVATE-memory-mapped file are visible to the process mapping the file is unspecified (they will not be written to the mapped file nor will they be visible to other processes mapping the same file).

However, we can verify if changes are actually visible on any given system / kernel version using this test program:

#include <string>
#include <iostream> 
#include <fstream> 
#include <sys/mman.h>
#include <sys/types.h>
#include <fcntl.h>
#include <unistd.h>
#include <assert.h>
#include <sys/stat.h>

using namespace std;

int main() {
    string filename = "deleteme.txt";
    // Create file
    ofstream fout(filename.c_str());
    fout << "ABCD";
    fout.close();
    // Create & open underlying file
    int fd = open(filename.c_str(), O_RDONLY);
    assert(fd != -1);
    // mmap with MAP_PRIVATE and allow writes to the mmapped pages
    char* mmappedData = (char*)mmap(NULL, 4, PROT_WRITE, MAP_PRIVATE | MAP_POPULATE, fd, 0);
    assert(mmappedData != MAP_FAILED);
    // Insert data into the MAP_PRIVATE area
    mmappedData[0] = 0;
    mmappedData[1] = 1;
    mmappedData[2] = 2;
    mmappedData[3] = 3;
    // Overwrite once more
    mmappedData[0] = 3;
    mmappedData[1] = 2;
    mmappedData[2] = 1;
    mmappedData[3] = 0;
    // Check data
    if(mmappedData[0] == 3 &&
        mmappedData[1] == 2 &&
        mmappedData[2] == 1 &&
        mmappedData[3] == 0) {
        cout << "Congrats, MAP_PRIVATE changes are reflected in memory!" << endl;
    } else {
        cout << "Nope, MAP_PRIVATE changes are NOT reflected in memory!" << endl;
    }
}

Download, compile and run using:

wget https://techoverflow.net/scripts/mmap-private-check.cpp && g++ -o mmap-private-check mmap-private-check.cpp && ./mmap-private-check

This will either print

Congrats, MAP_PRIVATE changes are reflected in memory!

(I only got this result for every Linux system I tested on, e.g. on Ubuntu 18.04) or

Nope, MAP_PRIVATE changes are NOT reflected in memory!

Note that this program only tests the visibility of the changes directly after writing the data. In principle, the Kernel is allowed to just discard your changes at any later point in time. So note that you are living on the edge here, doing changes to MAP_PRIVATE memory is not inherently safe but in practice it often works very well.

Apr

2019

How to fix std::wcout printing question marks (?) on Linux

Problem:

You are trying to print a wstring from a wstring literal using std::wcout (with an UTF-8-encoded source file):

wstring w = L"Test: äöü";
wcout << w << endl;

but when you run this program, you see

Test: ???

Solution:

Use setlocale() to set a UTF-8 locale:

setlocale( LC_ALL, "en_US.utf8" );
wstring w = L"Test: äöü";
wcout << w << endl;

This will print

Test: äöü

as expected.

Full example

#include <string>
#include <iostream>

using namespace std;

int main() {
    setlocale( LC_ALL, "en_US.utf8" );
    wstring w = L"Test: äöü";
    wcout << w << endl;
}

Compile like this:

g++ -o main main.cpp

Apr

2019

How to install RapidXML on Ubuntu

If you want to use RapidXML to parse XML data on Ubuntu, run this command:

sudo apt install librapidxml-dev -y

Since RapidXML is a header-only library, this will install the header files and no native libraries.

Apr

2019

How to free space by cleaning up old systemd / journald logs

If you need more free hard drive space on your system, you can clear old logs

Important note: When you’ve deleted old logs, they are deleted and there is no way to view them any more! Ensure that you don’t need them any more

View size occupied by those logs:

sudo du -sh /var/log/journal/

Tell journald to always only keep the last 25 Megabytes of logs (plus the current, not-rotated-yet logs):

sudo journalctl --vacuum-size=25M

Move current logs to archive (“rotate“) and only keep the last 25 Megabytes of logs:

sudo journalctl --flush --rotate
sudo journalctl --vacuum-size=25M

Alternatively you can tell it to keep only the last hour of logs (plus the current, not-rotated-yet logs):

sudo journalctl --vacuum-time=1h

Apr

2019

How to fix mount: unknown filesystem type ‘smbfs’

Problem:

When you’re trying to mount a Windows network share using a command like

sudo mount -t smbfs //Asus/store_n_go /mnt/

you see this error message:

mount: unknown filesystem type 'smbfs'

Solution:

First ensure samba is installed

sudo apt install samba

then try again using cifs as filesystem type instead of smbfs:

sudo mount -t cifs //Asus/store_n_go /mnt/

Apr

2019

How to print ISO8601 date using ‘date’ on command line

Use one of these commands to print the ISO8601 time using the date tool included in most Linux distributions.

date -I # 2019-04-06
date -Iseconds # 2019-04-06T17:22:49+02:00
date -Ins # 2019-04-06T17:23:08,505995625+02:00

For reference see the date manpage.

Apr

2019

How to install curl on Ubuntu Linux

On most Ubuntu installations, curl is already installed. In order to check, type

curl

into your shell and press return.

If you see a message like

curl: try 'curl --help' or 'curl --manual' for more information

curl is already installed and you don’t need to do anything.

In case you see this message:

Command 'curl' not found, but can be installed with:

apt install curl
Please ask your administrator.

or (more rarely) this message:

-bash: /usr/bin/curl: No such file or directory

you can install curl by copying

sudo apt install curl

into your shell and pressing return.

Apr

2019

How to upload your Python package to PyPI in 30 seconds

Prerequisite: Install twine:

sudo pip3 install twine

Before the next step, ensure you have no uncommitted files, because those will be deleted!

Also, ensure that your package is ready for release. Ensure that you have the correct version listed in setup.py

sudo chown -R $USER: .
git clean -xdf
python3 setup.py sdist
twine upload dist/*

Variant if you don’t have python3:

sudo chown -R $USER: .
git clean -xdf
python setup.py sdist
twine upload dist/*

For more detailed instructions see this post.

Detailed explanation of the commands:

sudo chown -R $USER: . Fix permission issues possibly introduced by sudo python3 setup.py install
git clean -xdf Remove uncommitted files and other fuzz
python3 setup.py sdist Build source package
twine upload dist/* This will ask you for user PyPI username and password and then upload the package.

Apr

2019

How to backup all indices from ElasticSearch

You can use elasticdump to backup all indices from your ElasticSearch cluster. Install using

sudo npm install elasticdump -g

If you don’t have npm, see How to install NodeJS 10.x on Ubuntu in 1 minute.

This package installs two binarys: elasticdump (used to dump a single index) and multielasticdump (used to dump multiple indices in parallel)

We can use multielasticdump to dump all indexes:

mkdir -p es_backup
multielasticdump --direction=dump --input=http://localhost:9200 --output=es_backup

Restore using:

multielasticdump --direction=load --input=es_backup --output=http://localhost:9200

Apr

2019

How to fix OpenVPN ‘failed to find GID for group openvpn’

Problem:

In your OpenVPN server logs you see this error message

failed to find GID for group openvpn

followed by a server restart (Exiting due to fatal error).

Solution:

Run this command to add the OpenVPN group:

sudo groupadd openvpn

In most cases, you’ll see this in your server log after doing that:

failed to find UID for user openvpn
Exiting due to fatal error

In that case, refer to our previous post on How to fix OpenVPN “failed to find UID for user openvpn”

Apr

2019

How to fix OpenVPN ‘failed to find UID for user openvpn’

Problem:

In your OpenVPN server logs you see this error message

failed to find UID for user openvpn

followed by a server restart (Exiting due to fatal error).

Solution:

Run this command to add the openvpn user and add that user to the openvpn group:

sudo useradd openvpn -g openvpn

Apr

2019

How to generate Diffie-Hellman (DH) parameters using OpenSSL

Problem:

For our webserver or VPN server, you want to use unique Diffie-Hellman parameters but you don’t know how to generate the .pem file using OpenSSL.

Solution:

Use this command to generate the parameters and save them in dhparams.pem:

openssl dhparam -out dhparams.pem 4096

This command generates Diffie-Hellman parameters with 4096 bits. This provides good security while still providing a very reasonable performance for modern devices. Depending on your preferred level of Paranoia you might want to increase the number of bits even more.

Note that even for “only” 4096 bits generating the parameters will usually take a couple of minutes. Larger parameter sizes might take many hours to days to generate. Ensure that you are generating the parameters on a fast computer and not on your Raspberry Pi or similar!

Apr

2019

How to enable SSH on Raspbian without a screen

You can open the boot partition on the SD card (the FAT32 partition) and create an empty file named ssh in the root directory of that partition. Ensure that the file is names ssh and not ssh.txt !

If you are in the correct working directory in the command line, use

touch ssh

On recent Ubuntu version, this will switch to the correct directory and create the file (but you need to mount the directory manually e.g. using your file explorer:

cd /media/$USER/boot && touch ssh

Don’t forget to unmount the boot drive before removing the SD card. Once you restart the Raspberry Pi with the modified SD card, SSH will be enabled without you having to attach a keyoard or screen to the Pi.

This approach was tested with the 2018-11-13 version of Raspbian and works with Raspberry Pi 1, Raspberry Pi 2 and Raspberry Pi 3.

Credits to Yahor for the original solution on StackOverflow!

Apr

2019

Fixing gcloud WARNING: `docker-credential-gcloud` not in system PATH

Problem:

You want to configure docker to be able to access Google Container Registry using

gcloud auth configure-docker

but you see this warning message:

WARNING: `docker-credential-gcloud` not in system PATH.
gcloud's Docker credential helper can be configured but it will not work until this is corrected.
gcloud credential helpers already registered correctly.

Solution:

Install docker-credential-gcloud using

sudo gcloud components install docker-credential-gcr

In case you see this error message:

ERROR: (gcloud.components.install) You cannot perform this action because this Cloud SDK installation is managed by an external package manager.
Please consider using a separate installation of the Cloud SDK created through the default mechanism described at: https://cloud.google.com/sdk/

use this alternate installation command instead (this command is for Linux, see the official documentation for other operating systems):

VERSION=1.5.0
OS=linux
ARCH=amd64

curl -fsSL "https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v${VERSION}/docker-credential-gcr_${OS}_${ARCH}-${VERSION}.tar.gz" \
  | tar xz --to-stdout ./docker-credential-gcr \
  | sudo tee /usr/bin/docker-credential-gcr > /dev/null && sudo chmod +x /usr/bin/docker-credential-gcr

After that, configure docker using

docker-credential-gcr configure-docker

Now you can retry running your original command.

For reference, see the official documentation.

Mar

2019

How to install kubectl on Ubuntu

Problem:

You want to run the Kubernetes kubectl command on Ubuntu but you see an error message like this:

command not found: kubectl

Solution:

Install kubectl using snap:

sudo snap install kubectl --classic

After this command has finished installing kubectl, in most cases you can use it immediately. In case you still get the command not found: kubectl error message, run $SHELL to reload your shell and check if /snap/bin is in your $PATH environment variable.

Mar

2019

How to fix Linux/Windows dual boot clock shift

Problem:

You have a dual-boot system. Every time you reboot from Linux to Windows, the time is shifted by several hours.

Solution:

On Linux, run

sudo timedatectl set-local-rtc 1

This will configure Linux to store local time in the RTC.

See this StackOverflow post for alternate solutions

Background:

Both Linux and Windows use the hardware clock (RTC – Real time clock) integrated into the computer hardware. However, Windows assumes that the RTC stores local time by default whereas Linux assumes the RTC stores UTC time.

Mar

2019

How to fix ModuleNotFoundError: No module named ‘grpc’ in Python

Problem:

You want to run a Python script that is using some Google Cloud services. However you see an error message similar to this:

[...]
  File "/usr/local/lib/python3.6/dist-packages/google/api_core/gapic_v1/__init__.py", line 16, in <module>
    from google.api_core.gapic_v1 import config
  File "/usr/local/lib/python3.6/dist-packages/google/api_core/gapic_v1/config.py", line 23, in <module>
    import grpc
ModuleNotFoundError: No module named 'grpc'

Solution:

Install the grpcio Python module:

sudo pip3 install grpcio

or, for Python 2.x

sudo pip install grpcio

Mar

2019

How to fix echo printing literal \n (backslash newline)

Problem:

You want to echo into a file like this:

echo "\ntest\n" > test.txt

but after doing that, test.txt contains the literal

\ntest\n

Solution:

Use echo -e (-e means: interpret backslash escapes):

echo -e "\ntest\n" > test.txt

After doing that, test.txt will contain test with a newline before and after.

Primary and supplemental groups

Problem:

Solution:

Full example

Problem:

Solution:

Detailed explanation of the commands:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Problem:

Solution:

Background:

Problem:

Solution:

Problem:

Solution:

Search

Categories