Linux

How to install NodeJS 12.x on Ubuntu in 1 minute

Run these shell commands on your Ubuntu computer to install NodeJS 12.x:

curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
sudo apt-get install -y nodejs

Instead of setup_12.x you can also choose other versions like setup_8.x. However, using this method, you can’t install multiple versions of NodeJS in parallel.

Source: Official nodesource documentation

Posted by Uli Köhler in Linux, NodeJS

What does ‘lsusb’ do on Linux?

lsusb lists all the USB devices to your computer.

Example output:

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 8087:0a2b Intel Corp. 
Bus 001 Device 002: ID 04f2:b5e8 Chicony Electronics Co., Ltd 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Each line represents a single USB device – but an USB device might also be internal to the computer and not connected to a physical USB port. For example, Bus 001 Device 003: ID 8087:0a2b Intel Corp. is an internal Bluetooth modem in my notebook.

Explanation:

  • Bus 001: The device is connected to the first USB bus (001) on the system
  • Device 003: In the aforementioned bus, the device has been assigned the ID 003.
  • ID 8087:0a2b: This is the ID assigned to that device. The first part (8087) is the Vendor ID (VID). The second part (0a2b) is the Product ID (PID). These IDs are assigned by the USB Implementer’s forum. This ID will always stay the same for any given device. If you buy a USB product multiple times, all of them will have the same ID.
  • Intel Corp.: This is the description of the device. Often Linux can show you more information, but in this case it’s just the name of the manufacturer.
Posted by Uli Köhler in Linux

What is a PPA on Linux?

A PPA (Personal Package Archive) is a repository where you can download packages, that is software, for Debian-based Linux distributions like Ubuntu.

In contrast to “normal” package source that are provided and maintained by distribution providers, PPAs can be created by anyone in order to publish one’s software or special versions of third-party software.

PPAs are hosted on Launchpad, a platform made and supported by Canonical, the company behind Ubuntu.

Are PPAs safe?

While in principle one could publish dangerous software on a PPA, in practice there is virtually no risk since:

  • Most often the PPA is provided by the original author of the software and can hence be trusted, if the software you want to install is trusted
  • PPAs are frequently used by amateurs and experts alike
  • You can only upload source DEB packages to the PPA which is built on Canonical’s server. This reduces the likelihood of smuggeling in binaries with malicious software.
  • Malicious PPAs are quickly removed once reported

Still it is good practice to keep in mind that installing software from anywhere – including PPAs – has a chance (although a very little one) to compromise your computer.

Posted by Uli Köhler in Linux

How to fix iotop ‘Netlink error: Operation not permitted (1)’

Problem:

You are trying to run iotop but you see an error message like

Netlink error: Operation not permitted (1)

The Linux kernel interfaces that iotop relies on now require root priviliges
or the NET_ADMIN capability. This change occured because a security issue
(CVE-2011-2494) was found that allows leakage of sensitive data across user
boundaries. If you require the ability to run iotop as a non-root user, please
configure sudo to allow you to run iotop as root.

Please do not file bugs on iotop about this.

Solution:

You need to run iotop as root:

sudo iotop

The reason for this is that iotop needs to access kernel interfaces that are only accessible to root users or if the user/process has the CAP_NET_ADMIN capability. In theory you could add this capability using sudo setcap cap_net_admin+ep /usr/sbin/iotop but on Ubuntu 18.04 this did not work for me.

 

Posted by Uli Köhler in Linux

How to use qemu with UEFI on Ubuntu

In order to use QEMU with UEFI on Ubuntu first install OVMF:

sudo apt -y install ovmf

After that, use the -bios /usr/share/ovmf/OVMF.fd with QEMU, for example:

sudo qemu-system-x86_64 -bios /usr/share/ovmf/OVMF.fd -hda /dev/sdb -m 1024
Posted by Uli Köhler in Linux

How to test your live USB without rebooting your computer on Linux

Problem:

You want to create a live USB stick e.g. gparted, Ubuntu etc and you quickly want to test it without rebooting your computer.

Solution:

You can use qemu, a system emulator, to do that.

First, install qemu. On Ubuntu:

sudo apt -y install qemu-system-x86

Now find out what the block  your USB stick is called (e.g. /dev/sdb). You can use lsblk to do this:

sudo lsblk

this will output, for example

[...]
sda      8:0    0 447,1G  0 disk 
├─sda1   8:1    0   549M  0 part 
├─sda2   8:2    0 121,2G  0 part 
├─sda3   8:3    0     1K  0 part 
└─sda5   8:5    0 325,4G  0 part /
sdb      8:16   1  29,3G  0 disk 
└─sdb1   8:17   1   268M  0 part

/dev/sda is usually your own hard drive (in my case, a 500 GB drive – 447.1G listed). In my case, I use a 32 GByte USB stick for the live USB, which is /dev/sdb (29.3 GB).

Note: You want to use the drive block device (e.g. /dev/sdb), not the partition block device (e.g. /dev/sdb1)

Now you can run qemu like this:

sudo qemu-system-x86_64 -hda /dev/sdb -m 1024

Remember to replace /dev/sdb by your USB stick block device (which we identified before using lsblk).

-m 1024 tells qemu to use 1024 Megabytes = 1 Gigabyte of RAM. You might need to increase this depending on what you are trying to boot.

Note that this approach will try to boot using BIOS and won’t try to boot UEFI-only drives like the ones created by tuxboot. In order to use UEFI with qemu you first need to install OVMF (which is a UEFI emulator that can be used for qemu):

sudo apt -y install ovmf

then run qemu like this:

sudo qemu-system-x86_64 -bios /usr/share/ovmf/OVMF.fd -hda /dev/sdb -m 1024

Remember to replace /dev/sdb by your USB stick block device (which we identified before using lsblk).

Since qemu is an emulator, it will be slower than booting your computer from the stick – but you can keep your computer running while you are testing your live USB stick.

Posted by Uli Köhler in Linux

How to install tuxboot on Ubuntu

According to the manual, you can install tuxboot using

sudo apt-add-repository -y ppa:thomas.tsai/ubuntu-tuxboot && sudo apt-get update && sudo apt-get install -y tuxboot

Now you can run tuxboot using

sudo tuxboot

 

Posted by Uli Köhler in Linux

How to fix raspivid ‘* failed to open vchiq instance’

Problem:

You want to capture a video using raspivid, e.g. using a command like

raspivid -o vid.h264

but you only see this error message:

* failed to open vchiq instance

with no video being captured.

Quick solution:

Run raspivid using sudo:

sudo raspivid -o vid.h264

Now either the video will be captured or you will see another error message if there is another issue with your camera setup.

Better solution:

Add your user to the video group so you don’t have to run raspivid as root:

sudo usermod -a -G video $USER

You only need to do this once. After that, log out and log back in (or close your SSH session and connect again). If in doubt, restart the Raspberry Pi. See What does ’sudo usermod -a -G group $USER‘ do on Linux? for more details.

Now you will be able to capture video as normal user:

raspivid -o vid.h264

 

Posted by Uli Köhler in Embedded, Linux

What does ‘sudo usermod -a -G group $USER’ do on Linux?

In our posts, especially posts like Solving Docker permission denied while trying to connect to the Docker daemon socket you can often see commands like

sudo usermod -a -G docker $USER

But what does this command actually do on your system?

Let’s break it down:

  • sudo means: Run this command as root. This is required for usermod since usually only root can modify which groups a user belongs to
  • usermod is a command that modifies the system configuration for a specific user ($USER in our example – see below). See the manpage documentation for more details on what you can do with it!
  • -a is a shortcut for --append: It means append the group to the list of groups the user belongs to!
  • -G is a shortcut for --groups: It tells usermod that the next argument is a group. Note that you need to use a capital -G here because we don’t want to modify the user’s primary group but the list of supplemental groups the user belongs to. See the Primary and supplemental groups section below for more details.
  • docker is the group we want to add $USER to. This could be any Linux group, provided that it exists. Use less /etc/group to have a look at all the groups that exist!
  • $USER is the user that we want to modify. $USER is a shell shortcut for the user that is running the command. This works even when using sudo (i.e. if your user is named uli and you are running sudo usermod -a -G docker $USER, the user uli will be added to the docker group, not the user root even though the command is run as root). You can also use a specific username instead of $USER, e.g. sudo usermod -a -G docker john to add the user john to the docker group

Primary and supplemental groups

When you browse through the usermod manpage, you’ll see there’s -G which adds a group to a user’s list of supplementary groups, and there’s -g which modifies a user’s primary group.

The pragmatic answer is: If you need to ask, you’ll always need to use -G.

Having to modify the primary group of a user is extremely rare in my experience. The purpose of primary groups existing is mainly that if you create a file, Linux needs to know which group it belongs to by default (i.e. if you don’t explicitly specify a group).

See this AskUbuntu post for more details on the purpose of primary and supplemental groups.

Posted by Uli Köhler in Linux

Are changes made to mmap MAP_PRIVATE visible to the current process?

The mmap mapage tells us that whether changes in memory made to a MAP_PRIVATE-memory-mapped file are visible to the process mapping the file is unspecified (they will not be written to the mapped file nor will they be visible to other processes mapping the same file).

However, we can verify if changes are actually visible on any given system / kernel version using this test program:

#include <string>
#include <iostream> 
#include <fstream> 
#include <sys/mman.h>
#include <sys/types.h>
#include <fcntl.h>
#include <unistd.h>
#include <assert.h>
#include <sys/stat.h>

using namespace std;

int main() {
    string filename = "deleteme.txt";
    // Create file
    ofstream fout(filename.c_str());
    fout << "ABCD";
    fout.close();
    // Create & open underlying file
    int fd = open(filename.c_str(), O_RDONLY);
    assert(fd != -1);
    // mmap with MAP_PRIVATE and allow writes to the mmapped pages
    char* mmappedData = (char*)mmap(NULL, 4, PROT_WRITE, MAP_PRIVATE | MAP_POPULATE, fd, 0);
    assert(mmappedData != MAP_FAILED);
    // Insert data into the MAP_PRIVATE area
    mmappedData[0] = 0;
    mmappedData[1] = 1;
    mmappedData[2] = 2;
    mmappedData[3] = 3;
    // Overwrite once more
    mmappedData[0] = 3;
    mmappedData[1] = 2;
    mmappedData[2] = 1;
    mmappedData[3] = 0;
    // Check data
    if(mmappedData[0] == 3 &&
        mmappedData[1] == 2 &&
        mmappedData[2] == 1 &&
        mmappedData[3] == 0) {
        cout << "Congrats, MAP_PRIVATE changes are reflected in memory!" << endl;
    } else {
        cout << "Nope, MAP_PRIVATE changes are NOT reflected in memory!" << endl;
    }
}

Download, compile and run using:

wget https://techoverflow.net/scripts/mmap-private-check.cpp && g++ -o mmap-private-check mmap-private-check.cpp && ./mmap-private-check

This will either print

Congrats, MAP_PRIVATE changes are reflected in memory!

(I only got this result for every Linux system I tested on, e.g. on Ubuntu 18.04) or

Nope, MAP_PRIVATE changes are NOT reflected in memory!

Note that this program only tests the visibility of the changes directly after writing the data. In principle, the Kernel is allowed to just discard your changes at any later point in time. So note that you are living on the edge here, doing changes to MAP_PRIVATE memory is not inherently safe but in practice it often works very well.

Posted by Uli Köhler in C/C++, Linux

How to fix std::wcout printing question marks (?) on Linux

Problem:

You are trying to print a wstring from a wstring literal using std::wcout (with an UTF-8-encoded source file):

wstring w = L"Test: äöü";
wcout << w << endl;

but when you run this program, you see

Test: ???

Solution:

Use setlocale() to set a UTF-8 locale:

setlocale( LC_ALL, "en_US.utf8" );
wstring w = L"Test: äöü";
wcout << w << endl;

This will print

Test: äöü

as expected.

Full example

#include <string>
#include <iostream>

using namespace std;

int main() {
    setlocale( LC_ALL, "en_US.utf8" );
    wstring w = L"Test: äöü";
    wcout << w << endl;
}

Compile like this:

g++ -o main main.cpp
Posted by Uli Köhler in C/C++, Linux

How to install RapidXML on Ubuntu

If you want to use RapidXML to parse XML data on Ubuntu, run this command:

sudo apt install librapidxml-dev -y

Since RapidXML is a header-only library, this will install the header files and no native libraries.

Posted by Uli Köhler in C/C++, Linux

How to free space by cleaning up old systemd / journald logs

If you need more free hard drive space on your system, you can clear old logs

Important note: When you’ve deleted old logs, they are deleted and there is no way to view them any more! Ensure that you don’t need them any more

View size occupied by those logs:

sudo du -sh /var/log/journal/

Tell journald to always only keep the last 25 Megabytes of logs (plus the current, not-rotated-yet logs):

sudo journalctl --vacuum-size=25M

Move current logs to archive (“rotate“) and only keep the last 25 Megabytes of logs:

sudo journalctl --flush --rotate
sudo journalctl --vacuum-size=25M

Alternatively you can tell it to keep only the last hour of logs (plus the current, not-rotated-yet logs):

sudo journalctl --vacuum-time=1h
Posted by Uli Köhler in Linux

How to fix mount: unknown filesystem type ‘smbfs’

Problem:

When you’re trying to mount a Windows network share using a command like

sudo mount -t smbfs //Asus/store_n_go /mnt/

you see this error message:

mount: unknown filesystem type 'smbfs'

Solution:

First ensure samba is installed

sudo apt install samba

then try again using cifs as filesystem type instead of smbfs:

sudo mount -t cifs //Asus/store_n_go /mnt/

 

Posted by Uli Köhler in Linux, Networking

How to print ISO8601 date using ‘date’ on command line

Use one of these commands to print the ISO8601 time using the date tool included in most Linux distributions.

date -I # 2019-04-06
date -Iseconds # 2019-04-06T17:22:49+02:00
date -Ins # 2019-04-06T17:23:08,505995625+02:00

For reference see the date manpage.

Posted by Uli Köhler in Linux

How to install curl on Ubuntu Linux

On most Ubuntu installations, curl is already installed. In order to check, type

curl

into your shell and press return.

If you see a message like

curl: try 'curl --help' or 'curl --manual' for more information

curl is already installed and you don’t need to do anything.

In case you see this message:

Command 'curl' not found, but can be installed with:

apt install curl
Please ask your administrator.

or (more rarely) this message:

-bash: /usr/bin/curl: No such file or directory

you can install curl by copying

sudo apt install curl

into your shell and pressing return.

Posted by Uli Köhler in Linux

How to upload your Python package to PyPI in 30 seconds

Prerequisite: Install twine:

sudo pip3 install twine

Before the next step, ensure you have no uncommitted files, because those will be deleted!

Also, ensure that your package is ready for release. Ensure that you have the correct version listed in setup.py

sudo chown -R $USER: .
git clean -xdf
python3 setup.py sdist
twine upload dist/*

Variant if you don’t have python3:

sudo chown -R $USER: .
git clean -xdf
python setup.py sdist
twine upload dist/*

For more detailed instructions see this post.

Detailed explanation of the commands:

  • sudo chown -R $USER: . Fix permission issues possibly introduced by sudo python3 setup.py install
  • git clean -xdf Remove uncommitted files and other fuzz
  • python3 setup.py sdist Build source package
  • twine upload dist/* This will ask you for user PyPI username and password and then upload the package.
Posted by Uli Köhler in Linux, Python

How to backup all indices from ElasticSearch

You can use elasticdump to backup all indices from your ElasticSearch cluster. Install using

sudo npm install elasticdump -g

If you don’t have npm, see How to install NodeJS 10.x on Ubuntu in 1 minute.

This package installs two binarys: elasticdump (used to dump a single index) and multielasticdump (used to dump multiple indices in parallel)

We can use multielasticdump to dump all indexes:

mkdir -p es_backup
multielasticdump --direction=dump --input=http://localhost:9200 --output=es_backup

Restore using:

multielasticdump --direction=load --input=es_backup --output=http://localhost:9200

 

Posted by Uli Köhler in ElasticSearch, Linux