How to fix git ‘error: cannot open .git/FETCH_HEAD: Permission denied’

Problem:

You want to run git pull or some other git command but you only see this error message:

error: cannot open .git/FETCH_HEAD: Permission denied

Solution:

This means that the .git directory is not owned by you. The easiest way to fix that is to change the owner of the directory to your user.

First, go to the root directory of the repository using cd, if you are not already there.

Then,

sudo chown -R $USER: .

In case you don’t have sudo access on that computer, the easiest way is to copy the repository to a directory where you have write access (e.g. using cp -r) or even clone the repository again.

Posted by Uli Köhler in Version management

ElasticSearch: How to iterate / scroll through all documents in index

In ElasticSearch, you can use the Scroll API to scroll through all documents in an entire index.

In Python you can scroll like this:

def es_iterate_all_documents(es, index, pagesize=250, scroll_timeout="1m", **kwargs):
    """
    Helper to iterate ALL values from a single index
    Yields all the documents.
    """
    is_first = True
    while True:
        # Scroll next
        if is_first: # Initialize scroll
            result = es.search(index=index, scroll="1m", **kwargs, body={
                "size": pagesize
            })
            is_first = False
        else:
            result = es.scroll(body={
                "scroll_id": scroll_id,
                "scroll": scroll_timeout
            })
        scroll_id = result["_scroll_id"]
        hits = result["hits"]["hits"]
        # Stop after no more docs
        if not hits:
            break
        # Yield each entry
        yield from (hit['_source'] for hit in hits)

This function will yield each document encountered in the index.

Example usage for index my_index:

es = Elasticsearch([{"host": "localhost"}])

for entry in es_iterate_all_documents(es, 'my_index'):
    print(entry) # Prints the document as stored in the DB

 

Posted by Uli Köhler in Databases, ElasticSearch, Python

How to build package or program using maven

In order to build a package using maven, run

mvn package

If you also want to install the maven package in the local maven repository (usually in ~/.m2), use

mvn install

In case you don’t have Maven installed, you can install it on Ubuntu using

sudo apt install maven

 

Posted by Uli Köhler in Build systems, Java

Which version of Windows 10 am I running? Find out in 15 seconds!

To find out which version & build of Windows 10 you are running, first press Windows key + R.
This will open a command prompt:

In that dialog, enter winver:

Now press Enter (also known as Return). This will open the version information window, for example:

In that window, you can immediately see the version & build you are running, highlighted in red:

In my case that is Version 1809 and Build 17763.475.

Posted by Uli Köhler in Windows

How to fix ‘cannot find symbol’ in Java

Problem:

You want to compile some Java source code (e.g. using Maven), but you get an error message like

[ERROR] /home/user/myproject/src/main/java/com/mydomain/myproject/Main.java:[123,40] cannot find symbol
[ERROR] symbol: class MyClass

Solution:

Java does not know where to find MyClass.

First, check if you have imported MyClass correctly.

If MyClass can be found in your library, you most likely are missing an import statement at the top of the file where the error occurs.

If, on the other hand, MyClass is imported from an external library, check if:

  • You have the correct version of the library. Possibly, you are using an old version of the library where MyClass is not present. This is often the case when using SNAPSHOT versions in maven since different developers might have different SNAPSHOTs so one might have issue building while another might not.
  • You are using the correct import statement (refer to the docs or the source code of the library to check if you are using the correct package.

If you don’t know in which library you can find a certain symbol in, see our post [To be done].

What exactly are ‘symbols’?

The concept and term of a symbol is used in many different programming languages.
Basically it means ‘a name that refers to something declared somewhere else in more detail’.

Therefore, if you encounter error messages like ‘cannot find symbol’, the compiler is trying to tell you: “I don’t know what that name refers to”.

Example:

When you declare a class in Java, e.g.

class MyClass {
    /* Your code goes here ! */
}

you can later refer to that class by its name, MyClass, e.g. in

MyClass class = new MyClass();

In that line of code, MyClass is used symbolically to refer to the full class declaration (class MyClass { /* ... */}) which we listed before.

Hence, MyClass is used as a symbol to refer to your previous (full) declaration of MyClass.

If the name MyClass, however, has no associated full declaration, the compiler will tell you ‘cannot find symbol’.

Posted by Uli Köhler in Java

What is a PPA on Linux?

A PPA (Personal Package Archive) is a repository where you can download packages, that is software, for Debian-based Linux distributions like Ubuntu.

In contrast to “normal” package source that are provided and maintained by distribution providers, PPAs can be created by anyone in order to publish one’s software or special versions of third-party software.

PPAs are hosted on Launchpad, a platform made and supported by Canonical, the company behind Ubuntu.

Are PPAs safe?

While in principle one could publish dangerous software on a PPA, in practice there is virtually no risk since:

  • Most often the PPA is provided by the original author of the software and can hence be trusted, if the software you want to install is trusted
  • PPAs are frequently used by amateurs and experts alike
  • You can only upload source DEB packages to the PPA which is built on Canonical’s server. This reduces the likelihood of smuggeling in binaries with malicious software.
  • Malicious PPAs are quickly removed once reported

Still it is good practice to keep in mind that installing software from anywhere – including PPAs – has a chance (although a very little one) to compromise your computer.

Posted by Uli Köhler in Linux

How to fix iotop ‘Netlink error: Operation not permitted (1)’

Problem:

You are trying to run iotop but you see an error message like

Netlink error: Operation not permitted (1)

The Linux kernel interfaces that iotop relies on now require root priviliges
or the NET_ADMIN capability. This change occured because a security issue
(CVE-2011-2494) was found that allows leakage of sensitive data across user
boundaries. If you require the ability to run iotop as a non-root user, please
configure sudo to allow you to run iotop as root.

Please do not file bugs on iotop about this.

Solution:

You need to run iotop as root:

sudo iotop

The reason for this is that iotop needs to access kernel interfaces that are only accessible to root users or if the user/process has the CAP_NET_ADMIN capability. In theory you could add this capability using sudo setcap cap_net_admin+ep /usr/sbin/iotop but on Ubuntu 18.04 this did not work for me.

 

Posted by Uli Köhler in Linux

How to use qemu with UEFI on Ubuntu

In order to use QEMU with UEFI on Ubuntu first install OVMF:

sudo apt -y install ovmf

After that, use the -bios /usr/share/ovmf/OVMF.fd with QEMU, for example:

sudo qemu-system-x86_64 -bios /usr/share/ovmf/OVMF.fd -hda /dev/sdb -m 1024
Posted by Uli Köhler in Linux

How to test your live USB without rebooting your computer on Linux

Problem:

You want to create a live USB stick e.g. gparted, Ubuntu etc and you quickly want to test it without rebooting your computer.

Solution:

You can use qemu, a system emulator, to do that.

First, install qemu. On Ubuntu:

sudo apt -y install qemu-system-x86

Now find out what the block  your USB stick is called (e.g. /dev/sdb). You can use lsblk to do this:

sudo lsblk

this will output, for example

[...]
sda      8:0    0 447,1G  0 disk 
├─sda1   8:1    0   549M  0 part 
├─sda2   8:2    0 121,2G  0 part 
├─sda3   8:3    0     1K  0 part 
└─sda5   8:5    0 325,4G  0 part /
sdb      8:16   1  29,3G  0 disk 
└─sdb1   8:17   1   268M  0 part

/dev/sda is usually your own hard drive (in my case, a 500 GB drive – 447.1G listed). In my case, I use a 32 GByte USB stick for the live USB, which is /dev/sdb (29.3 GB).

Note: You want to use the drive block device (e.g. /dev/sdb), not the partition block device (e.g. /dev/sdb1)

Now you can run qemu like this:

sudo qemu-system-x86_64 -hda /dev/sdb -m 1024

Remember to replace /dev/sdb by your USB stick block device (which we identified before using lsblk).

-m 1024 tells qemu to use 1024 Megabytes = 1 Gigabyte of RAM. You might need to increase this depending on what you are trying to boot.

Note that this approach will try to boot using BIOS and won’t try to boot UEFI-only drives like the ones created by tuxboot. In order to use UEFI with qemu you first need to install OVMF (which is a UEFI emulator that can be used for qemu):

sudo apt -y install ovmf

then run qemu like this:

sudo qemu-system-x86_64 -bios /usr/share/ovmf/OVMF.fd -hda /dev/sdb -m 1024

Remember to replace /dev/sdb by your USB stick block device (which we identified before using lsblk).

Since qemu is an emulator, it will be slower than booting your computer from the stick – but you can keep your computer running while you are testing your live USB stick.

Posted by Uli Köhler in Linux

How to install tuxboot on Ubuntu

According to the manual, you can install tuxboot using

sudo apt-add-repository -y ppa:thomas.tsai/ubuntu-tuxboot && sudo apt-get update && sudo apt-get install -y tuxboot

Now you can run tuxboot using

sudo tuxboot

 

Posted by Uli Köhler in Linux

How to fix raspivid ‘* failed to open vchiq instance’

Problem:

You want to capture a video using raspivid, e.g. using a command like

raspivid -o vid.h264

but you only see this error message:

* failed to open vchiq instance

with no video being captured.

Quick solution:

Run raspivid using sudo:

sudo raspivid -o vid.h264

Now either the video will be captured or you will see another error message if there is another issue with your camera setup.

Better solution:

Add your user to the video group so you don’t have to run raspivid as root:

sudo usermod -a -G video $USER

You only need to do this once. After that, log out and log back in (or close your SSH session and connect again). If in doubt, restart the Raspberry Pi. See What does ’sudo usermod -a -G group $USER‘ do on Linux? for more details.

Now you will be able to capture video as normal user:

raspivid -o vid.h264

 

Posted by Uli Köhler in Embedded, Linux

ElasticSearch: How to iterate all documents in index using Python (up to 10000 documents)

Important Note: This simple approach only works for up to ~10000 documents. Prefer using our scroll-based solution: See ElasticSearch: How to iterate / scroll through all documents in index

Use this helper function to iterate over all the documens in an index

def es_iterate_all_documents(es, index, pagesize=250, **kwargs):
    """
    Helper to iterate ALL values from
    Yields all the documents.
    """
    offset = 0
    while True:
        result = es.search(index=index, **kwargs, body={
            "size": pagesize,
            "from": offset
        })
        hits = result["hits"]["hits"]
        # Stop after no more docs
        if not hits:
            break
        # Yield each entry
        yield from (hit['_source'] for hit in hits)
        # Continue from there
        offset += pagesize

Usage example:

for entry in es_iterate_all_documents(es, 'my_index'):
    print(entry) # Prints the document as stored in the DB

How it works

You can iterate over all documents in an index in ElasticSearch by using queries like

{
    "size": 250,
    "from": 0
}

and increasing "from" by "size" after each iteration.

Posted by Uli Köhler in Databases, ElasticSearch, Python

Fixing ElasticSearch ‘Unknown key for a VALUE_NUMBER in [offset].’

The error message

Unknown key for a VALUE_NUMBER in [offset].

in ElasticSearch tells you that in the query JSON you have specified an offset (numeric value) but ElasticSearch doesn’t know what to do with offset.

This is easy to fix: In order to specify an offset to start from, use from instead of offset.

Incorrect:

{
    "size": 250,
    "offset": 1000
}

Correct:

{
    "size": 250,
    "from": 1000
}

 

Posted by Uli Köhler in Databases, ElasticSearch

Fixing ElasticSearch ‘Unknown key for a START_ARRAY in […].’

When you get an error message like

elasticsearch.exceptions.RequestError: RequestError(400, 'parsing_exception', 'Unknown key for a START_ARRAY in [size].')

in ElasticSearch, look for the value in the [brackets]. In our example this is [size].

Your query contains an array for that key but an array is not allowed.

For example, this query is malformed:

{
    "size": [10, 11]
}

because size takes a number, not an array.

In Python, if you are programmatically building your array, you might have an extra comma at the end of your line.

For example, if you have a line like

query["size"] = 250,

this is just syntactic sugar for

query["size"] = (250,)

i.e. it will set size to a tuple with one element. In the JSON this will result in

{
    "size": [250]
}

In order to fix that issue, remove the comma from the end of the line

query["size"] = 250

which will result in the correct query JSON

{
    "size": 250
}
Posted by Uli Köhler in Databases, ElasticSearch

ElasticSearch equivalent to MongoDB .count()

The ElasticSearch equivalent to MongoDB’s count() is also called count. It can be used in a similar way

When you have an ElasticSearch query like (example in Python)

result = es.search(index="my_index", body={
    "query": {
        "match": {
            "my_field": "my_value"
        }
    }
})
result_docs = [hit["_source"] for hit in result["hits"]["hits"]]

you can easily change it to a count-only query by replacing search with count:

result = es.count(index="my_index", body={
    "query": {
        "match": {
            "my_field": "my_value"
        }
    }
})
result_count = result["count"] # e.g. 200
Posted by Uli Köhler in Databases, ElasticSearch

Fixing ElasticSearch ‘no [query] registered for [query]’

Problem:

You want to run a query in ElasticSearch, but you get an error message like

elasticsearch.exceptions.RequestError: RequestError(400, 'parsing_exception', 'no [query] registered for [query]')

Solution:

In your query body, you have two "query" objects nested in each other. Remove the outer "query", keeping only the inner one.

Example:

Incorrect:

{
    "query": {
        "query": {
            "match": {
                "my_field": "my_value"
            }
        }
    }
}

Correct:

{
    "query": {
        "match": {
            "my_field": "my_value"
        }
    }
}

 

Posted by Uli Köhler in Databases, ElasticSearch

How to fix ‘ng upgrade’ error ‘The specified command (“upgrade”) is invalid’

Problem:

You want to run ng upgrade to update your angular project libraries, but you see this error message:

The specified command ("upgrade") is invalid. For a list of available options,
run "ng help".

Did you mean "update"?

Solution:

You need to run

ng update

instead of ng upgrade (which is not a valid command).

This is what Did you mean "update"? in the error message is intended to point you towards.

Posted by Uli Köhler in Angular, Javascript

How to fix ElasticSearch ‘Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true’

Problem:

You want to create a mapping in ElasticSearch but you see an error message like

elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true.')

Solution:

As already suggested in the error message, set the include_type_name parameter to True.

With the Python API this is as simple as adding include_type_name=True to the put_mapping(...) call:

es.indices.put_mapping(index='my_index', body=my_mapping, doc_type='_doc', include_type_name=True)

In case you now see an error like

TypeError: put_mapping() got an unexpected keyword argument 'include_type_name'

you need to upgrade your elasticsearch python library, e.g. using

sudo pip3 install --upgrade elasticsearch

 

Posted by Uli Köhler in Databases, ElasticSearch, Python

How to view ElasticSearch cluster health using curl

To view the cluster health of your ElasticSearch cluster use

curl -X GET "http://localhost:9200/_cluster/health?pretty=true"

If your ElasticSearch is not running on localhost, replace localhost by the hostname or IP address ElasticSearch is running on.

Example output:
{
  "cluster_name" : "docker-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

The most important information from this is:

  • "cluster_name" : "docker-cluster" The name you assigned to your cluster. You should verify that you are connecting to the correct cluster. All ElasticSearch nodes from that cluster must have the same cluster name, or they won’t connect!
  • "number_of_nodes" : 1 The number of nodes currently in the cluster. Sometimes some nodes take longer to start up, so if there are some nodes missing, wait a minute and retry
  • "status" : "green" The status or cluster health of your cluster.

The cluster health can take three values:

  • green: Everything is OK with your cluster (like in our example)
  • yellow: Your cluster is mostly OK, but some shards couldn’t be replicated. This is often the case with cluster consisting of one node only (in that case, note that a data loss on the one node can not be recovered)
  • red: Something is wrong with the cluster. Usually that’s some configuration issue, so be sure to check the logs.

Also see the official reference on cluster health

If you are looking for help on how to setup your ElasticSearch cluster using docker and docker-compose, you can generate your config file using our generator at ElasticSearch docker-compose.yml and systemd service generator.

Posted by Uli Köhler in Databases, ElasticSearch

What does ‘sudo usermod -a -G group $USER’ do on Linux?

In our posts, especially posts like Solving Docker permission denied while trying to connect to the Docker daemon socket you can often see commands like

sudo usermod -a -G docker $USER

But what does this command actually do on your system?

Let’s break it down:

  • sudo means: Run this command as root. This is required for usermod since usually only root can modify which groups a user belongs to
  • usermod is a command that modifies the system configuration for a specific user ($USER in our example – see below). See the manpage documentation for more details on what you can do with it!
  • -a is a shortcut for --append: It means append the group to the list of groups the user belongs to!
  • -G is a shortcut for --groups: It tells usermod that the next argument is a group. Note that you need to use a capital -G here because we don’t want to modify the user’s primary group but the list of supplemental groups the user belongs to. See the Primary and supplemental groups section below for more details.
  • docker is the group we want to add $USER to. This could be any Linux group, provided that it exists. Use less /etc/group to have a look at all the groups that exist!
  • $USER is the user that we want to modify. $USER is a shell shortcut for the user that is running the command. This works even when using sudo (i.e. if your user is named uli and you are running sudo usermod -a -G docker $USER, the user uli will be added to the docker group, not the user root even though the command is run as root). You can also use a specific username instead of $USER, e.g. sudo usermod -a -G docker john to add the user john to the docker group

Primary and supplemental groups

When you browse through the usermod manpage, you’ll see there’s -G which adds a group to a user’s list of supplementary groups, and there’s -g which modifies a user’s primary group.

The pragmatic answer is: If you need to ask, you’ll always need to use -G.

Having to modify the primary group of a user is extremely rare in my experience. The purpose of primary groups existing is mainly that if you create a file, Linux needs to know which group it belongs to by default (i.e. if you don’t explicitly specify a group).

See this AskUbuntu post for more details on the purpose of primary and supplemental groups.

Posted by Uli Köhler in Linux