How to fix ElasticSearch ‘[match] query doesn’t support multiple fields, found […] and […]’

Problem:

You want to run an ElasticSearch query like

{
    "query": {
        "match" : {
            "one_field" : "one_value",
            "another_field": "another_value"
        }
    }
}

but you only see an error message like

elasticsearch.exceptions.RequestError: RequestError(400, 'parsing_exception', "[match] query doesn't support multiple fields, found [one_field] and [another_field]")

Solution:

Match queries only support one field. You should use a bool query with a must clause containing multiple match queries instead:

{
    "query": {
        "bool": {
            "must": [
                {"match": {"one_field" : "one_value"}},
                {"match": {"another_field" : "another_value"}},
            ]
        }
    }
}

Also see the official docs for the MultiMatch query.

How to fix ElasticSearch ‘no [query] registered for [missing]’

Problem:

You are trying to run an ElasticSearch query like

{
    "query": {
        "missing" : { "field" : "myfield" }
    }
}

to find documents that do not have myfield.

However you only see an error message like this:

elasticsearch.exceptions.RequestError: RequestError(400, 'parsing_exception', 'no [query] registered for [missing]')

Solution:

As the ElasticSearch documentation tells, us, there is no missing query! You instead need to use an exists query inside a must_not clause:

{
    "query": {
        "bool": {
            "must_not": {
                "exists": {
                    "field": "myfield"
                }
            }
        }
    }
}

 

How to fix Google Cloud Build ignoring .dockerignore

Problem:

You want to run a docker image build on Google Cloud Build, but the client is trying to upload a huge context image to Google Cloud even though you have added all your large directories to your .dockerignore and the build works fine locally.

Solution:

Google Cloud Build ignores .dockerignore by design – the equivalent is called .gcloudignore.

You can copy the .dockerignore behaviour for gcloud by running

cp .dockerignore .gcloudignore

 

Are changes made to mmap MAP_PRIVATE visible to the current process?

The mmap mapage tells us that whether changes in memory made to a MAP_PRIVATE-memory-mapped file are visible to the process mapping the file is unspecified (they will not be written to the mapped file nor will they be visible to other processes mapping the same file).

However, we can verify if changes are actually visible on any given system / kernel version using this test program:

#include <string>
#include <iostream> 
#include <fstream> 
#include <sys/mman.h>
#include <sys/types.h>
#include <fcntl.h>
#include <unistd.h>
#include <assert.h>
#include <sys/stat.h>

using namespace std;

int main() {
    string filename = "deleteme.txt";
    // Create file
    ofstream fout(filename.c_str());
    fout << "ABCD";
    fout.close();
    // Create & open underlying file
    int fd = open(filename.c_str(), O_RDONLY);
    assert(fd != -1);
    // mmap with MAP_PRIVATE and allow writes to the mmapped pages
    char* mmappedData = (char*)mmap(NULL, 4, PROT_WRITE, MAP_PRIVATE | MAP_POPULATE, fd, 0);
    assert(mmappedData != MAP_FAILED);
    // Insert data into the MAP_PRIVATE area
    mmappedData[0] = 0;
    mmappedData[1] = 1;
    mmappedData[2] = 2;
    mmappedData[3] = 3;
    // Overwrite once more
    mmappedData[0] = 3;
    mmappedData[1] = 2;
    mmappedData[2] = 1;
    mmappedData[3] = 0;
    // Check data
    if(mmappedData[0] == 3 &&
        mmappedData[1] == 2 &&
        mmappedData[2] == 1 &&
        mmappedData[3] == 0) {
        cout << "Congrats, MAP_PRIVATE changes are reflected in memory!" << endl;
    } else {
        cout << "Nope, MAP_PRIVATE changes are NOT reflected in memory!" << endl;
    }
}

Download, compile and run using:

wget https://techoverflow.net/scripts/mmap-private-check.cpp && g++ -o mmap-private-check mmap-private-check.cpp && ./mmap-private-check

This will either print

Congrats, MAP_PRIVATE changes are reflected in memory!

(I only got this result for every Linux system I tested on, e.g. on Ubuntu 18.04) or

Nope, MAP_PRIVATE changes are NOT reflected in memory!

Note that this program only tests the visibility of the changes directly after writing the data. In principle, the Kernel is allowed to just discard your changes at any later point in time. So note that you are living on the edge here, doing changes to MAP_PRIVATE memory is not inherently safe but in practice it often works very well.

How to get file size using boost::filesystem library

To get the filesize (in bytes) of any file (test.xml in our example) using the boost::filesystem library, use this snippet:

#include <experimental/filesystem>
#include <iostream>

using namespace std;
using namespace std::experimental::filesystem;

int main() {
    size_t filesize = file_size("test.xml");
    cout << filesize << endl;
}

You need to link the boost_filesystem and the boost_system libraries, i.e. compile like this:

g++ -o test test.cpp -lboost_filesystem -lboost_system

How to get file size using C++17 filesystem library

To get the filesize (in bytes) of any file (test.xml in our example) using only the C++17 filesystem library, use this snippet:

#include <experimental/filesystem>
#include <iostream>

using namespace std;
using namespace std::experimental::filesystem;

int main() {
    size_t filesize = file_size("test.xml");
    cout << filesize << endl;
}

With GCC/G++ you need to link the stdc++fs library, i.e. compile like this:

g++ -o test test.cpp -lstdc++fs

How to fix GCC undefined reference to std::experimental::filesystem::…

Problem:

You want to compile an executable using GCC/G++ that uses functionality from std::experimental::filesystem but you get an error message like this during compilation / linking:

test.cpp:(.text+0x33): undefined reference to `std::experimental::filesystem::v1::file_size(std::experimental::filesystem::v1::__cxx11::path const&)'
/tmp/ccSXeCVb.o: In function `std::experimental::filesystem::v1::__cxx11::path::path<char [9], std::experimental::filesystem::v1::__cxx11::path>(char const (&) [9])':
test.cpp:(.text._ZNSt12experimental10filesystem2v17__cxx114pathC2IA9_cS3_EERKT_[_ZNSt12experimental10filesystem2v17__cxx114pathC5IA9_cS3_EERKT_]+0x73): undefined reference to `std::experimental::filesystem::v1::__cxx11::path::_M_split_cmpts()'
collect2: error: ld returned 1 exit status

Solution:

You need to link the stdc++fs library, i.e. append -lstdc++fs to your g++ command.

For example, a working command looks like this:

g++ -o myprogram main.cpp -lstdc++fs

Minimal RapidXML file reader example

XML:

<?xml version="1.0" encoding="UTF-8"?>
<root-element>Test text</root-element>

C++:

#include <rapidxml/rapidxml_utils.hpp>
#include <iostream> 

using namespace rapidxml;
using namespace std;

int main() {
    file<> xmlFile("test.xml");
    // Create & parse document
    xml_document<> doc;
    doc.parse<0>(xmlFile.data());
    // Get root node
    xml_node<> *root = doc.first_node("root-element");
    // Print content
    if(root == nullptr) {
        cerr << "Element not found!" << endl;
    } else { // We found the root element
        cout << root->value() << endl;
    }
}

 

How to fix std::wcout printing question marks (?) on Linux

Problem:

You are trying to print a wstring from a wstring literal using std::wcout (with an UTF-8-encoded source file):

wstring w = L"Test: äöü";
wcout << w << endl;

but when you run this program, you see

Test: ???

Solution:

Use setlocale() to set a UTF-8 locale:

setlocale( LC_ALL, "en_US.utf8" );
wstring w = L"Test: äöü";
wcout << w << endl;

This will print

Test: äöü

as expected.

How to cout a wstring or wchar_t in C++

Problem:

You have a std::wstring that you want to print using cout

wstring w = L"Test: äöü";
cout << w << endl;

but you see a long error message that ends like this:

/usr/include/c++/7/ostream:682:5: note:   template argument deduction/substitution failed:
/usr/include/c++/7/ostream: In substitution of 'template<class _Ostream, class _Tp> typename std::enable_if<std::__and_<std::__not_<std::is_lvalue_reference<_Tp> >, std::__is_convertible_to_basic_ostream<_Ostream>, std::__is_insertable<typename std::__is_convertible_to_basic_ostream<_Tp>::__ostream_type, const _Tp&, void> >::value, typename std::__is_convertible_to_basic_ostream<_Tp>::__ostream_type>::type std::operator<<(_Ostream&&, const _Tp&) [with _Ostream = std::basic_ostream<char>&; _Tp = std::__cxx11::basic_string<wchar_t>]':
test.cpp:9:13:   required from here
/usr/include/c++/7/ostream:682:5: error: no type named 'type' in 'struct std::enable_if<false, std::basic_ostream<char>&>'

Solution:

You need to use std::wcout instead of std::cout:

wstring w = L"Test: äöü";
wcout << w << endl

 

How to get attribute value in RapidXML

Problem:

You have a rapidxml::xml_node instance for which you want to access a specific attribute,  e.g. my-attribute

Solution:

Use first_attribute with the name argument set to a string:

rapidxml::xml_attribute<>* attr = node->first_attribute("my-attribute");

Remember that first_attribute() returns nullptr if no such attribute exists so be sure to check for that to avoid segmentation faults!

Full example:

XML:

<?xml version="1.0" encoding="UTF-8"?>
<root-element>
    <child my-attr="foo"></child>
</root-element>

C++:

#include <rapidxml/rapidxml_utils.hpp>
#include <string>
#include <iostream> 

using namespace rapidxml;
using namespace std;

int main() {
    rapidxml::file<> xmlFile("test.xml");
    // Create & parse document
    rapidxml::xml_document<> doc;
    doc.parse<0>(xmlFile.data());
    
    // Get root node
    rapidxml::xml_node<> *root = doc.first_node("root-element");
    rapidxml::xml_node<> *child = root->first_node("child");

    // Get & print attribute
    rapidxml::xml_attribute<>* attr = child->first_attribute("my-attr");
    if(attr == nullptr) {
        cout << "No such attribute!" << endl;
    } else {
        cout << attr->value() << endl;
    }
}

Or you can use this snippet function to get either the attribute value or a default value:

#include <rapidxml/rapidxml_utils.hpp>
#include <string>
#include <iostream> 

using namespace rapidxml;
using namespace std;

/**
 * Return either the ->value() of attr or default_value if attr == nullptr
 */
inline string attr_value_or_default(rapidxml::xml_attribute<>* attr, string default_value="") {
    if(attr == nullptr) {
        return default_value;
    } else {
        return attr->value();
    }
}

int main() {
    rapidxml::file<> xmlFile("test.xml");
    // Create & parse document
    rapidxml::xml_document<> doc;
    doc.parse<0>(xmlFile.data());
    
    // Get root node
    rapidxml::xml_node<> *root = doc.first_node("root-element");
    rapidxml::xml_node<> *child = root->first_node("child");

    // Get & print attribute+
    rapidxml::xml_attribute<>* attr = child->first_attribute("my-attr");
    cout << attr_value_or_default(attr, "No such attribute!") << endl;
}

How to fix fatal error: rapidxml_utils.hpp: No such file or directory

If you are seeing an error message like fatal error: rapidxml_utils.hpp: No such file or directory in a line like

#include "rapidxml_utils.hpp"

or

#include <rapidxml_utils.hpp>

you likely need to replace that line by

#include <rapidxml/rapidxml_utils.hpp>

If this also doesn’t work, you might not have RapidXML installed. See How to install RapidXML on Ubuntu for details on how to do that on Ubuntu.

How to fix Python bottle Unsupported response type: <class ‘dict’>

Problem:

You are running a Python HTTP server using bottle. When you access your HTTP server endpoint, you see a HTTP 500 error message like

Unsupported response type: <class 'dict'>

Solution:

This occurs because you are trying to return a Python list of dictionaries, for example in

from bottle import route, run, template, response

@route('/')
def index():
    # We expect bottle to return a JSON here
    # but that doesn't happen!
    return [{"a": "b"}]

run(host='localhost', port=8080)

In order to work around this behaviour, you need to set response.content_type and explicitly use json.dumps() to convert your JSON into a string:

from bottle import route, run, template, response
import json

@route('/')
def index():
    response.content_type = 'application/json'
    return json.dumps([{"a": "b"}])

run(host='localhost', port=8080)

 

How to fix ElasticSearch Root mapping definition has unsupported parameters: [mappings : {properties=…

Problem:

You want to create an ElasticSearch index with a custom mapping or update the mapping of an existing ElasticSearch index but you see an error message like

elasticsearch.exceptions.RequestError: RequestError(400, 'mapper_parsing_exception', 'Root mapping definition has unsupported parameters:  [mappings : {properties={num_total={type=integer}, approved={type=integer}, num_translated={type=integer}, pattern_length={type=integer}, num_unapproved={type=integer}, pattern={type=keyword}, num_approved={type=integer}, translated={type=integer}, untranslated={type=integer}, num_untranslated={type=integer}, group={type=keyword}}}]')

Solution:

This can point to multiple issues. Essentially, ElasticSearch is trying to tell you that the structure of your JSON is not correct.

Often this error is misinterpreted as individual field definitions being wrong, but this is rarely the issue (and only if an individual field definition is completely malformed).

If your message is structured like

... unsupported parameters:  [mappings : ...

then the most likely root cause is that you have mappings nested inside mappings in your JSON. This also applies if you update a mapping (put_mapping) – in this case the outer mapping is implicit!

Example: Your code looks like this:

es.indices.put_mapping(index='my_index, doc_type='_doc', body={
    "mappings": {
        "properties": {
            "pattern": {
                "type":  "keyword"
            }
        }
    }
})

ElasticSearch will internally create a JSON like this internally:

{
    "mappings": {
        "mappings": {
            "properties": {
                "pattern": {
                    "type":  "keyword"
                }
            }
        }
    }
}

See that there are two mappings inside each other? ElasticSearch does not view this as a correctly structured JSON, therefore you need to remove the "mapping": {...} from your code, resulting in

es.indices.put_mapping(index='my_index, doc_type='_doc', body={
    "properties": {
        "pattern": {
            "type":  "keyword"
        }
    }
})

Fixing ElasticSearch ‘No handler for type [int] declared on field …’

Problem:

You want to create an index with a custom mapping in ElasticSearch but you see an error message like this:

elasticsearch.exceptions.RequestError: RequestError(400, 'mapper_parsing_exception', 'No handler for type [int] declared on field [id]')

Solution:

You likely have a mapping like

"id": {
    "type":  "int"
}

in your mapping properties.

The issue here is int: ElasticSearch uses integer as type of integers, not int!

In order to fix the issue, change the property to

"id": {
    "type":  "integer"
}

and retry creating the index.

 

How to free space by cleaning up old systemd / journald logs

If you need more free hard drive space on your system, you can clear old logs

Important note: When you’ve deleted old logs, they are deleted and there is no way to view them any more! Ensure that you don’t need them any more

View size occupied by those logs:

sudo du -sh /var/log/journal/

Tell journald to always only keep the last 25 Megabytes of logs (plus the current, not-rotated-yet logs):

sudo journalctl --vacuum-size=25M

Move current logs to archive (“rotate“) and only keep the last 25 Megabytes of logs:

sudo journalctl --flush --rotate
sudo journalctl --vacuum-size=25M

Alternatively you can tell it to keep only the last hour of logs (plus the current, not-rotated-yet logs):

sudo journalctl --vacuum-time=1h