What does ‘sudo usermod -a -G group $USER’ do on Linux?

In our posts, especially posts like Solving Docker permission denied while trying to connect to the Docker daemon socket you can often see commands like

sudo usermod -a -G docker $USER

But what does this command actually do on your system?

Let’s break it down:

  • sudo means: Run this command as root. This is required for usermod since usually only root can modify which groups a user belongs to
  • usermod is a command that modifies the system configuration for a specific user ($USER in our example – see below). See the manpage documentation for more details on what you can do with it!
  • -a is a shortcut for --append: It means append the group to the list of groups the user belongs to!
  • -G is a shortcut for --groups: It tells usermod that the next argument is a group. Note that you need to use a capital -G here because we don’t want to modify the user’s primary group but the list of supplemental groups the user belongs to. See the Primary and supplemental groups section below for more details.
  • docker is the group we want to add $USER to. This could be any Linux group, provided that it exists. Use less /etc/group to have a look at all the groups that exist!
  • $USER is the user that we want to modify. $USER is a shell shortcut for the user that is running the command. This works even when using sudo (i.e. if your user is named uli and you are running sudo usermod -a -G docker $USER, the user uli will be added to the docker group, not the user root even though the command is run as root). You can also use a specific username instead of $USER, e.g. sudo usermod -a -G docker john to add the user john to the docker group

Primary and supplemental groups

When you browse through the usermod manpage, you’ll see there’s -G which adds a group to a user’s list of supplementary groups, and there’s -g which modifies a user’s primary group.

The pragmatic answer is: If you need to ask, you’ll always need to use -G.

Having to modify the primary group of a user is extremely rare in my experience. The purpose of primary groups existing is mainly that if you create a file, Linux needs to know which group it belongs to by default (i.e. if you don’t explicitly specify a group).

See this AskUbuntu post for more details on the purpose of primary and supplemental groups.

Posted by Uli Köhler in Linux

Fixing docker ‘unable to delete …- image is being used by running container’

Problem:

You want to delete a docker image using a command like

docker image rm c91b419ac445

but you see an error message like

Error response from daemon: conflict: unable to delete c91b419ac445 (cannot be forced) - image is being used by running container 3477a4dcdce2

Solution:

There is currently a container running that uses the image you are trying to delete. We will solve this issue by first stopping the container and then deleting the image

Warning: Deleting the image is dangerous since you cannot undo deleting the image ! Also note that force-stopping a running container might result in data loss if that container is doing something important !

Run these commands to stop the container and delete the image:

# Force-stop the container
docker container rm --force <container ID>
# Delete the image
docker image rm <image ID>

Copy <container ID> from the end of your original error message (3477a4dcdce2 in my example).

Copy <image ID> from the beginning of your error message. This is the same image ID you originally intended to delete (c91b419ac445) in my example.

In my example, the command would be

# Force-stop the container
docker container rm --force 3477a4dcdce2
# Delete the image
docker image rm c91b419ac445

Note that there might be multiple containers running using this image, so if you keep getting a similar error message, you might need to repeat this command.

Background information:

Docker will not allow you to force-delete the image using

docker image rm c91b419ac445 --force

as you can also see from the (cannot be forced) clause of your original error message. This behaviour makes sense since the container would crash in an undefineable manner if the underlying image is deleted.

Note that we could use docker image rm --force after stopping the container but this is typically not required and might result in additional risks for other containers, e.g. if other images depend on said image. Since docker uses layered images. Read the background information section of our post Docker: Remove all images and containers to learn more about how docker images work from an image management perspective.

Posted by Uli Köhler in Container, Docker

How to get path to the wp_content directory in WordPress plugin

Problem:

You are writing a wordpress plugin in which you need the path to the wp-content directory on the filesystem.

Solution:

Use the WP_CONTENT_DIR constant.

$path_to_wp_content = WP_CONTENT_DIR; // e.g. "/var/sites/techoverflow.net/wp-content"

Note that WP_CONTENT_DIR has no trailing slash.

Use WP_CONTENT_DIR . "/" like this

$path_to_wp_content = WP_CONTENT_DIR . "/"; // e.g. "/var/sites/techoverflow.net/wp-content/"

to get the path to wp_content including a trailing slash.

Posted by Uli Köhler in PHP, Wordpress

How to fix WP_Query not listing any posts

Problem:

In your custom WordPress plugin or theme you have code like

$query = new WP_Query( );
while ( $query->have_posts() ) {
        $query->the_post();
        // ... your code to process the post ...
}

But $query->have_posts() is always false and the loop is never executed.

Solution:

WP_Query works only if you use an appropriate query. Using no query at all is not equivalent to list all posts!

This is likely what you want to do:

$query = new WP_Query(array('post_type' => 'post'));
Posted by Uli Köhler in PHP, Wordpress

How to recursively delete directory using C++17 filesystem library

To remove a file or directory (for example my-directory) use remove_all from the C++17 filesystem library:

remove_all("my-directory");

This will remove my-directory and all its sub-directories and files recursively.

Full example:

#include <experimental/filesystem>
using namespace std::experimental::filesystem;

int main() {
    remove_all("my-directory");
}

In case you are using GCC, you need to compile the file like this:

g++ -o delete-cpp17 delete-cpp17.cpp -lstdc++fs

You need to link the stdc++fs library so the functions from the C++17 filesystem library are available to your program.

If you just want to remove a file and don’t want to risk deleting an entire directory tree, use remove instead of remove_all or see our previous post How to delete file using C++17 filesystem library

Posted by Uli Köhler in C/C++

How to delete file using C++17 filesystem library

To remove a file (for example test.txt) use remove from the C++17 filesystem library:

remove("test.txt");

Full example:

#include <experimental/filesystem>
using namespace std::experimental::filesystem;

int main() {
    remove("test.txt");
}

In case you are using GCC, you need to compile the file like this:

g++ -o delete-cpp17 delete-cpp17.cpp -lstdc++fs

You need to link the stdc++fs library so the functions from the C++17 filesystem library are available to your program.

Note that remove does not recursively remove directories! Use remove_all or see How to recursively delete directory using C++17 filesystem library

Posted by Uli Köhler in C/C++

Is it a file or directory? Using C++17 filesystem library

You can use is_regular_file to check any path (either a C++17 path object or just a string). Similarly you can use is_directory to check if the given path belongs to a directory.

Note that these just return false if the file or directory does not exist!

// Check if something is a file
bool isTestTxtAFile = is_regular_file("test.txt"); // true
bool isMyDirectoryAFile = is_regular_file("my-directory"); // false
bool isDoesNotExistAFile = is_regular_file("does-not-exist"); // false
// Check if something is a directory
bool isTestTxtADirectory = is_directory("test.txt"); // false
bool isMyDirectoryADirectory = is_directory("my-directory"); // true
bool isDoesNotExistADirectory = is_directory("does-not-exist"); // false
Posted by Uli Köhler in C/C++

How to recursively list a directory using C++17 filesystem library

Note: To find out how to list a directory non-recursively, just replace recursive_directory_iterator by directory_iterator or see our full post How to list a directory using C++17 filesystem library.

To recursively list a directory using the C++17 filesystem library use this snippet:

#include <experimental/filesystem>

using namespace std::experimental::filesystem;

for(const directory_entry& entry : recursive_directory_iterator("my-directory")) {
    const auto& path = entry.path();
    // ...
}

Full example:

#include <iostream>
#include <string>
#include <experimental/filesystem>

using namespace std;
using namespace std::experimental::filesystem;

int main() {
    for(const directory_entry& entry : recursive_directory_iterator("my-directory")) {
        // Is it a file / directory?
        bool isNormalFile = is_regular_file(entry);
        bool isDirectory = is_directory(entry);

        auto path = entry.path();
        // Path: my-directory/test.txt
        string pathString = path.string();

        // Filename: test.txt
        string filenameString = path.filename().string();

        // Extension: txt
        string extensionString = path.extension().string();

        // NOTE: You can also "cout << path" directly
    }
}

Compile like this:

g++ -o filesystem-example filesystem-example.cpp -lstdc++fs

For this directory:

my-directory
├── subdirectory
│   └── subdir-test.txt
└── test.txt

this will list subdirectory , subdirectory/subdir-test.txt and test.txt.

Posted by Uli Köhler in C/C++

How to list a directory using C++17 filesystem library

To list a directory (non-recursively) using the C++17 filesystem library use this snippet:

#include <experimental/filesystem>

using namespace std::experimental::filesystem;

for(const directory_entry& entry : directory_iterator("my-directory")) {
    const auto& path = entry.path();
    // ...
}

Full example:

#include <iostream>
#include <string>
#include <experimental/filesystem>

using namespace std;
using namespace std::experimental::filesystem;

int main() {
    for(const directory_entry& entry : directory_iterator("my-directory")) {
        // Is it a file / directory?
        bool isNormalFile = is_regular_file(entry);
        bool isDirectory = is_directory(entry);

        auto path = entry.path();
        // Path: my-directory/test.txt
        string pathString = path.string();

        // Filename: test.txt
        string filenameString = path.filename().string();

        // Extension: txt
        string extensionString = path.extension().string();

        // NOTE: You can also "cout << path" directly
    }
}

Compile like this:

g++ -o filesystem-example filesystem-example.cpp -lstdc++fs

For this directory:

my-directory
├── subdirectory
│   └── subdir-test.txt
└── test.txt

this will only list subdirectory and test.txt

Posted by Uli Köhler in C/C++

How to fix ElasticSearch ‘[match] query doesn’t support multiple fields, found […] and […]’

Problem:

You want to run an ElasticSearch query like

{
    "query": {
        "match" : {
            "one_field" : "one_value",
            "another_field": "another_value"
        }
    }
}

but you only see an error message like

elasticsearch.exceptions.RequestError: RequestError(400, 'parsing_exception', "[match] query doesn't support multiple fields, found [one_field] and [another_field]")

Solution:

Match queries only support one field. You should use a bool query with a must clause containing multiple match queries instead:

{
    "query": {
        "bool": {
            "must": [
                {"match": {"one_field" : "one_value"}},
                {"match": {"another_field" : "another_value"}},
            ]
        }
    }
}

Also see the official docs for the MultiMatch query.

Posted by Uli Köhler in Databases, ElasticSearch

How to fix ElasticSearch ‘no [query] registered for [missing]’

Problem:

You are trying to run an ElasticSearch query like

{
    "query": {
        "missing" : { "field" : "myfield" }
    }
}

to find documents that do not have myfield.

However you only see an error message like this:

elasticsearch.exceptions.RequestError: RequestError(400, 'parsing_exception', 'no [query] registered for [missing]')

Solution:

As the ElasticSearch documentation tells, us, there is no missing query! You instead need to use an exists query inside a must_not clause:

{
    "query": {
        "bool": {
            "must_not": {
                "exists": {
                    "field": "myfield"
                }
            }
        }
    }
}

 

Posted by Uli Köhler in Databases, ElasticSearch

How to fix Google Cloud Build ignoring .dockerignore

Problem:

You want to run a docker image build on Google Cloud Build, but the client is trying to upload a huge context image to Google Cloud even though you have added all your large directories to your .dockerignore and the build works fine locally.

Solution:

Google Cloud Build ignores .dockerignore by design – the equivalent is called .gcloudignore.

You can copy the .dockerignore behaviour for gcloud by running

cp .dockerignore .gcloudignore

 

Posted by Uli Köhler in Cloud, Container, Docker

Are changes made to mmap MAP_PRIVATE visible to the current process?

The mmap mapage tells us that whether changes in memory made to a MAP_PRIVATE-memory-mapped file are visible to the process mapping the file is unspecified (they will not be written to the mapped file nor will they be visible to other processes mapping the same file).

However, we can verify if changes are actually visible on any given system / kernel version using this test program:

#include <string>
#include <iostream> 
#include <fstream> 
#include <sys/mman.h>
#include <sys/types.h>
#include <fcntl.h>
#include <unistd.h>
#include <assert.h>
#include <sys/stat.h>

using namespace std;

int main() {
    string filename = "deleteme.txt";
    // Create file
    ofstream fout(filename.c_str());
    fout << "ABCD";
    fout.close();
    // Create & open underlying file
    int fd = open(filename.c_str(), O_RDONLY);
    assert(fd != -1);
    // mmap with MAP_PRIVATE and allow writes to the mmapped pages
    char* mmappedData = (char*)mmap(NULL, 4, PROT_WRITE, MAP_PRIVATE | MAP_POPULATE, fd, 0);
    assert(mmappedData != MAP_FAILED);
    // Insert data into the MAP_PRIVATE area
    mmappedData[0] = 0;
    mmappedData[1] = 1;
    mmappedData[2] = 2;
    mmappedData[3] = 3;
    // Overwrite once more
    mmappedData[0] = 3;
    mmappedData[1] = 2;
    mmappedData[2] = 1;
    mmappedData[3] = 0;
    // Check data
    if(mmappedData[0] == 3 &&
        mmappedData[1] == 2 &&
        mmappedData[2] == 1 &&
        mmappedData[3] == 0) {
        cout << "Congrats, MAP_PRIVATE changes are reflected in memory!" << endl;
    } else {
        cout << "Nope, MAP_PRIVATE changes are NOT reflected in memory!" << endl;
    }
}

Download, compile and run using:

wget https://techoverflow.net/scripts/mmap-private-check.cpp && g++ -o mmap-private-check mmap-private-check.cpp && ./mmap-private-check

This will either print

Congrats, MAP_PRIVATE changes are reflected in memory!

(I only got this result for every Linux system I tested on, e.g. on Ubuntu 18.04) or

Nope, MAP_PRIVATE changes are NOT reflected in memory!

Note that this program only tests the visibility of the changes directly after writing the data. In principle, the Kernel is allowed to just discard your changes at any later point in time. So note that you are living on the edge here, doing changes to MAP_PRIVATE memory is not inherently safe but in practice it often works very well.

Posted by Uli Köhler in C/C++, Linux

How to get file size using boost::filesystem library

To get the filesize (in bytes) of any file (test.xml in our example) using the boost::filesystem library, use this snippet:

#include <experimental/filesystem>
#include <iostream>

using namespace std;
using namespace std::experimental::filesystem;

int main() {
    size_t filesize = file_size("test.xml");
    cout << filesize << endl;
}

You need to link the boost_filesystem and the boost_system libraries, i.e. compile like this:

g++ -o test test.cpp -lboost_filesystem -lboost_system
Posted by Uli Köhler in C/C++

How to get file size using C++17 filesystem library

To get the filesize (in bytes) of any file (test.xml in our example) using only the C++17 filesystem library, use this snippet:

#include <experimental/filesystem>
#include <iostream>

using namespace std;
using namespace std::experimental::filesystem;

int main() {
    size_t filesize = file_size("test.xml");
    cout << filesize << endl;
}

With GCC/G++ you need to link the stdc++fs library, i.e. compile like this:

g++ -o test test.cpp -lstdc++fs
Posted by Uli Köhler in C/C++

How to fix GCC undefined reference to std::experimental::filesystem::…

Problem:

You want to compile an executable using GCC/G++ that uses functionality from std::experimental::filesystem but you get an error message like this during compilation / linking:

test.cpp:(.text+0x33): undefined reference to `std::experimental::filesystem::v1::file_size(std::experimental::filesystem::v1::__cxx11::path const&)'
/tmp/ccSXeCVb.o: In function `std::experimental::filesystem::v1::__cxx11::path::path<char [9], std::experimental::filesystem::v1::__cxx11::path>(char const (&) [9])':
test.cpp:(.text._ZNSt12experimental10filesystem2v17__cxx114pathC2IA9_cS3_EERKT_[_ZNSt12experimental10filesystem2v17__cxx114pathC5IA9_cS3_EERKT_]+0x73): undefined reference to `std::experimental::filesystem::v1::__cxx11::path::_M_split_cmpts()'
collect2: error: ld returned 1 exit status

Solution:

You need to link the stdc++fs library, i.e. append -lstdc++fs to your g++ command.

For example, a working command looks like this:

g++ -o myprogram main.cpp -lstdc++fs
Posted by Uli Köhler in C/C++, GCC errors

Minimal RapidXML file reader example

XML:

<?xml version="1.0" encoding="UTF-8"?>
<root-element>Test text</root-element>

C++:

#include <rapidxml/rapidxml_utils.hpp>
#include <iostream> 

using namespace rapidxml;
using namespace std;

int main() {
    file<> xmlFile("test.xml");
    // Create & parse document
    xml_document<> doc;
    doc.parse<0>(xmlFile.data());
    // Get root node
    xml_node<> *root = doc.first_node("root-element");
    // Print content
    if(root == nullptr) {
        cerr << "Element not found!" << endl;
    } else { // We found the root element
        cout << root->value() << endl;
    }
}

Build configuration

add_executable(rapidxml-example rapidxml-example.cpp)

Compile using

cmake .
make
Posted by Uli Köhler in C/C++

How to fix std::wcout printing question marks (?) on Linux

Problem:

You are trying to print a wstring from a wstring literal using std::wcout (with an UTF-8-encoded source file):

wstring w = L"Test: äöü";
wcout << w << endl;

but when you run this program, you see

Test: ???

Solution:

Use setlocale() to set a UTF-8 locale:

setlocale( LC_ALL, "en_US.utf8" );
wstring w = L"Test: äöü";
wcout << w << endl;

This will print

Test: äöü

as expected.

Full example

#include <string>
#include <iostream>

using namespace std;

int main() {
    setlocale( LC_ALL, "en_US.utf8" );
    wstring w = L"Test: äöü";
    wcout << w << endl;
}

Compile like this:

g++ -o main main.cpp
Posted by Uli Köhler in C/C++, Linux