Python threading minimal example

This minimal example shows how to create a thread that prints Hello world every second:

import threading
import time

def my_thread_func():
    while True:
        print("Hello world")
        time.sleep(1)

my_thread = threading.Thread(target=my_thread_func)
my_thread.start()

 

Posted by Uli Köhler in Python

How to fix wifi blocked on boot on Raspberry Pi 4

After migrating a fresh Raspbian install using the official 2020-04 Raspbian light image from my Raspberry Pi Model 2 to my new Raspberry Model 4, the Wifi was disabled at boot.

I tried configuring the Wifi using raspi-config but that didn’t change anything.

First, try rfkill unblock all and then reboot in order to check if the Wifi adapter is still unblocked after the reboot. In my case, this fixed the issue permanently and wifi worked immediately.

If that doesn’t help, check if country=... is set in /etc/wpa_supplicant/wpa_supplicant.conf. You need to set it to your correct country code to comply with regulatory limits. For example, use country=DE to set the regulatory domain to Germany.

Posted by Uli Köhler in Linux, Raspberry Pi

How to setup OnlyOffice using docker-compose & nginx

Prerequisite: Install docker and docker-compose

For example, follow our guide How to install docker and docker-compose on Ubuntu in 30 seconds

Step 1: Create docker-compose.yml

Create the directory where we’ll install OnlyOffice using

sudo mkdir /var/lib/onlyoffice

and then edit the docker-compose configuration using e.g.

sudo nano /var/lib/onlyoffice/docker-compose.yml

and copy and paste this content

version: '3'
services:
  onlyoffice-documentserver:
    image: onlyoffice/documentserver:latest
    restart: always
    environment:
      - JWT_ENABLED=true
      - JWT_SECRET=ahSaTh4waeKe4zoocohngaihaub5pu
    ports:
      - 2291:80
    volumes:
      - ./onlyoffice/data:/var/www/onlyoffice/Data
      - ./onlyoffice/lib:/var/lib/onlyoffice
      - ./onlyoffice/logs:/var/log/onlyoffice
      - ./onlyoffice/db:/var/lib/postgresql

Now add your custom password in JWT_SECRET=... ! Don’t forget this step, or anyone can use your OnlyOffice server ! I’m using pwgen 30 to generate a new random password (install using sudo apt -y install pwgen).

Step 2: Setup systemd service

Create the service using sudo nano /etc/systemd/system/onlyoffice.service:

[Unit]
Description=OnlyOffice server
Requires=docker.service
After=docker.service

[Service]
Restart=always
User=root
Group=docker
# Shutdown container (if running) when unit is stopped
ExecStartPre=/usr/local/bin/docker-compose -f /var/lib/onlyoffice/docker-compose.yml down -v
# Start container when unit is started
ExecStart=/usr/local/bin/docker-compose -f /var/lib/onlyoffice/docker-compose.yml up
# Stop container when unit is stopped
ExecStop=/usr/local/bin/docker-compose -f /var/lib/onlyoffice/docker-compose.yml down -v

[Install]
WantedBy=multi-user.target

Now enable & start the service using

sudo systemctl enable onlyoffice
sudo systemctl start onlyoffice

Step 3:  Create nginx reverse proxy configuration

Note that we mapped OnlyOffice’s port 80 to port 2291. In case you’re not using nginx as reverse proxy, you need to manually configure your reverse proxy to pass requests to port 2291.

server {
    server_name onlyoffice.mydomain.org;

    access_log /var/log/nginx/onlyoffice.access_log;
    error_log /var/log/nginx/onlyoffice.error_log info;

    location / {
        proxy_pass http://127.0.0.1:2291;
        proxy_http_version 1.1;
        proxy_read_timeout 3600s;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host            $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        add_header X-Frontend-Host $host;
        # Uncomment this line and reload once you have setup TLS for that domain !
        # add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    }

    listen 80;
}

Now test if your nginx config works using nginx -t and reload using service nginx reload.

Now I recommend to setup Let’s Encrypt for your domain so that your OnlyOffice instance will only be accessed using an encrypted connecting (sudo certbot --nginx, see other guides if you don’t know how to do that).

Once certbot asks you whether to redirect, choose option 2 – Redirect to HTTPS.

Step 4: Test OnlyOffice

If your installation worked, you should see a screen like this:

If not, try checking the logs using

sudo journalctl -xu onlyoffice

(Optional) Step 5: Configure NextCloud to use OnlyOffice

If you are running NextCloud, go to Settings => ONLYOFFICE and enter your domain and the JWT_SECRET you created before:

Ensure that Connect to demo ONLYOFFICE Document Server is unchecked and click Save.

Nextcloud will tell you at the top right if it has been able to connect to your OnlyOffice instance successfully:

  • Settings successfully updated means that NextCloud is now connected to OnlyOffice
  • Invalid token means that your password / secret key does not match
  • Other messages typically mean that your OnlyOffice is not running or that you haven’t entered the correct domain or protocol. I recommend to only use https:// – use http:// for testing only and don’t forget to revert back to https:// once you have found the issue.
Posted by Uli Köhler in Container, Docker, Linux, nginx

How to fix RCurl ‘Cannot find curl-config’ or ‘checking for curl-config… no’

Problem:

You want to install RCurl using

Rscript -e "install.packages('RCurl')"

but you see an error message like

Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL 'https://cloud.r-project.org/src/contrib/RCurl_1.98-1.2.tar.gz'
Content type 'application/x-gzip' length 699583 bytes (683 KB)
==================================================
downloaded 683 KB

* installing *source* package ‘RCurl’ ...
** package ‘RCurl’ successfully unpacked and MD5 sums checked
** using staged installation
checking for curl-config... no
Cannot find curl-config
ERROR: configuration failed for package ‘RCurl’
* removing ‘/usr/local/lib/R/site-library/RCurl’

The downloaded source packages are in
 ‘/tmp/RtmpSVQKd1/downloaded_packages’
Warning message:
In install.packages("RCurl") :
  installation of package ‘RCurl’ had non-zero exit status

Solution:

You need to install the curl development headers on your machine. RCurl complains that it can’t find those development headers.

On Ubuntu/debian, use

sudo apt -y install libcurl4-openssl-dev

 

Posted by Uli Köhler in R

How to open a shell in an LXC container

You can run a shell in your LXC container using

lxc exec [name of container] /bin/bash

for example

lxc exec mycontainer /bin/bash
Posted by Uli Köhler in LXC

How to fix LXC ‘Error: The remote isn’t a private LXD server’

Problem:

You are trying to launch a LXC container using a command like

lxc launch mycontainer ubuntu:18.04

but you see this error message:

Solution:

Your command line arguments are in the wrong order. You need to run lxc launch [image] [name of container], not lxc launch [name of container] [image] ! The correct command looks like this:

lxc launch ubuntu:18.04 mycontainer

 

Posted by Uli Köhler in LXC

How to fix LXC file push ‘Error: Path already exists as a directory: File too large’

Problem:

You are trying to copy a file to a LXC container using lxc file push, but you see this error message:

Error: Path already exists as a directory: File too large

Solution:

Add a slash (/) at the end of your path, for example:

mycontainer/root => mycontainer/root/

Working example:

lxc file push myfile.zip mycontainer/root/

Also see our previous post How to copy files to a LXC container

Posted by Uli Köhler in LXC

How to copy files to a LXC container

Once you’ve created a LXC container using a command like

lxc launch ubuntu:18.04 mycontainer

you can push files to the container using

lxc file push myfile.zip mycontainer/root/

This will copy the local file myfile.zip to /root/myfile.zip on the container. Ensure that your path ends with /, since lxc file push myfile.zip mycontainer/root  will show this error message:

Error: Path already exists as a directory: File too large

In that case, add a slash (/) to the end of your destination path (e.g. mycontainer/root => mycontainer/root/).

Posted by Uli Köhler in Container, LXC

Pressure in a rotating body of liquid calculator

This online calculator allows you to calculate the pressure of a rotating body of liquid with a given density at a specified radius. Note that gravity is not accounted for and the pressure is relative to the pressure at r=0. See this forum post for more details on the formula.

TechOverflow calculators:
You can enter values with SI suffixes like 12.2m (equivalent to 0.012) or 14k (14000) or 32u (0.000032).
The results are calculated while you type and shown directly below the calculator, so there is no need to press return or click on a Calculate button. Just make sure that all inputs are green by entering valid values.

kg/m³

Hz

m



\omega = 2\cdot\pi\cdot n[Hz]
P = \frac{1}{2}\cdot\rho\cdot\omega²\cdot r²

Posted by Uli Köhler in Calculators

How to fix Python unittest __init__() takes 1 positional argument but 2 were given

Problem:

You are trying to run your Python unit tests using the unittest package, but you see this unspecific stack trace:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.7/unittest/__main__.py", line 18, in <module>
    main(module=None)
  File "/usr/lib/python3.7/unittest/main.py", line 100, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python3.7/unittest/main.py", line 124, in parseArgs
    self._do_discovery(argv[2:])
  File "/usr/lib/python3.7/unittest/main.py", line 244, in _do_discovery
    self.createTests(from_discovery=True, Loader=Loader)
  File "/usr/lib/python3.7/unittest/main.py", line 154, in createTests
    self.test = loader.discover(self.start, self.pattern, self.top)
  File "/usr/lib/python3.7/unittest/loader.py", line 349, in discover
    tests = list(self._find_tests(start_dir, pattern))
  File "/usr/lib/python3.7/unittest/loader.py", line 414, in _find_tests
    yield from self._find_tests(full_path, pattern, namespace)
  File "/usr/lib/python3.7/unittest/loader.py", line 406, in _find_tests
    full_path, pattern, namespace)
  File "/usr/lib/python3.7/unittest/loader.py", line 460, in _find_test_path
    return self.loadTestsFromModule(module, pattern=pattern), False
  File "/usr/lib/python3.7/unittest/loader.py", line 124, in loadTestsFromModule
    tests.append(self.loadTestsFromTestCase(obj))
  File "/usr/lib/python3.7/unittest/loader.py", line 93, in loadTestsFromTestCase
    loaded_suite = self.suiteClass(map(testCaseClass, testCaseNames))
  File "/usr/lib/python3.7/unittest/suite.py", line 24, in __init__
    self.addTests(tests)
  File "/usr/lib/python3.7/unittest/suite.py", line 57, in addTests
    for test in tests:
TypeError: __init__() takes 1 positional argument but 2 were given

Solution:

You have at least one test case like this one:

class MyTest(unittest.TestCase):
    def __init__(self):
        self.x = 1.0

    def test_stuff(self):
        assert(self.x == 1.0)

Overriding __init__(...) is not possible in this way when using unittest. You need to use setUp() instead.

Usually, just replacing def __init__(self): by def setUp(self): will do the trick. unittests will call setUp() automatically.

Our example will look like this:

class MyTest(unittest.TestCase):
    def setUp(self):
        self.x = 1.0

    def test_stuff(self):
        assert(self.x == 1.0)

If the error still persists, check if you have more testcases overriding the __init__() method.

Posted by Uli Köhler in Python

How to fix Terraria does not start / immediately exits on Linux

If your Terraria exits immediately and the Terrria window never appears, try

cd ~/.local/share/Steam/steamapps/common/Terraria
./Terraria.bin.x86_64 > terraria.log

As it turns out, Terraria only starts properly on my machine if you redirect stdout to a file (or pipe it into another program), hence > terraria.log is neccessary to get it running.

Posted by Uli Köhler in Linux

Hollow cylinder volume calculator (from diameters & height)

TechOverflow calculators:
You can enter values with SI suffixes like 12.2m (equivalent to 0.012) or 14k (14000) or 32u (0.000032).
The results are calculated while you type and shown directly below the calculator, so there is no need to press return or click on a Calculate button. Just make sure that all inputs are green by entering valid values.

m

m

m



V = \pi\cdot(\frac{OD}{2})²\cdot h\ -\ \pi\cdot(\frac{ID}{2})²\cdot h

Posted by Uli Köhler in Calculators

Cylinder volume calculator (from diameter & height)

Calculate the volume of a cylinder from its diameter and height using this online calculator – formula included.

TechOverflow calculators:
You can enter values with SI suffixes like 12.2m (equivalent to 0.012) or 14k (14000) or 32u (0.000032).
The results are calculated while you type and shown directly below the calculator, so there is no need to press return or click on a Calculate button. Just make sure that all inputs are green by entering valid values.

m

m



V = \pi\cdot(\frac{d}{2})²\cdot h

Posted by Uli Köhler in Allgemein

Centrifuge acceleration calculator (from rpm and diameter)

Calculate the acceleration from its rotational speed in RPM and its diameter using this online calculator – formula included.

TechOverflow calculators:
You can enter values with SI suffixes like 12.2m (equivalent to 0.012) or 14k (14000) or 32u (0.000032).
The results are calculated while you type and shown directly below the calculator, so there is no need to press return or click on a Calculate button. Just make sure that all inputs are green by entering valid values.

rpm

m





n[rps] = \frac{n[rpm]}{60\frac{s}{min}}a[m/s²] = 4\cdot\pi^2\cdot(\frac{d}{2})\cdot(n[rps])²a[g] = \frac{a[m/s²]}{9.80665\frac{m/s²}{g}}

Posted by Uli Köhler in Calculators

Centrifuge diameter calculator (from acceleration & rpm)

TechOverflow calculators:
You can enter values with SI suffixes like 12.2m (equivalent to 0.012) or 14k (14000) or 32u (0.000032).
The results are calculated while you type and shown directly below the calculator, so there is no need to press return or click on a Calculate button. Just make sure that all inputs are green by entering valid values.

rpm

g




a[m/s²] = a[g] \cdot 9.80665\frac{m/s²}{g}n[rps] = \frac{n[rpm]}{60\frac{s}{min}}d_{centrifuge} = 2\cdot\frac{a[m/s²]}{4\cdot\pi^2\cdot(n[rps])²}

Posted by Uli Köhler in Calculators

Two easy ways to download a file using Python requests

requests does not provide a

Option 1: Use requests_download:

First, install requests_download using

sudo pip3 install requests_download

or equivalent.

Now you can use it like this:

from requests_download import download

download(url, filename)

It also has built-in progress bar support:

from requests_download import download, HashTracker, ProgressTracker
from progressbar import DataTransferBar # sudo pip3 install progressbar2

progress = ProgressTracker(DataTransferBar())

download(pdfUrl, filename, trackers=(progress,))

Option 2: Do it yourself:

Use this snippet in your code:

import requests
import shutil

def requests_download_file(url, filename):
    with requests.get(url, stream=True) as response:
        with open(filename, 'wb') as fout:
            shutil.copyfileobj(response.raw, fout)
Posted by Uli Köhler in Python

How I reduced gitlab memory consumption in my docker-based setup

I’m currently running 4 separate dockerized gitlab instances on my server. These tend to consume quite a lot of memory even when not being used for some time.

Reduce the number of unicorn worker processes

The gitlab default is to use 6 unicorn worker processes. By reducing the number of workers to 2, my gitlab memory consumption decreased by approximately 60%:

unicorn['worker_processes'] = 2

In my dockerized setup, I justed updated the GITLAB_OMNIBUS_CONFIG in docker-compose.yml and restarted the instance. If you didn’t install gitlab using docker, you might need to sudo gitlab-ctl reconfigure.

Note that you need at least 2 unicorn workers for gitlab to work properly. See this issue for details.

Also note that reducing the number of workers to the minimum will likely impact your gitlab performance in a negative way. Increase the number of workers if you notice a lack in performance.

Disable Prometheus monitoring

Most small installation do not need Prometheus, the monitoring tool integrated into Gitlab:

prometheus_monitoring['enable'] = false

Reduce sidekiq concurrency

sidekiq is the background job processor integrated into Gitlab. The default concurrency is 25. I recommend reducing it.

sidekiq['concurrency'] = 2

This might cause background jobs to take longer since they have to wait in queue, but for small installations it does not matter in my experience.

Reduce the PostgreSQL shared memory

This was recommended on StackOverflow.

postgresql['shared_buffers'] = "256MB"

Setting this too low might cause a heavier IO load and all operations (including website page loads) might be slower.

The complete config

This is the configuration (combined from all strategies listed above) in order to get down the memory consumption:

# Unicorn config
unicorn['worker_processes'] = 2
# PostgreSQL config
postgresql['shared_buffers'] = "256MB"
# Sidekiq config
sidekiq['concurrency'] = 2
# Prometheus config
prometheus_monitoring['enable'] = false

 

Posted by Uli Köhler in Docker, git

How to fix ElasticSearch docker AccessDeniedException[/usr/share/elasticsearch/data/nodes];”,

Problem:

You are trying to start a dockerized ElasticSearch instance but you see an error log like

lasticsearch1    | {"type": "server", "timestamp": "2020-04-18T01:17:27,564Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "docker-cluster", "node.name": "elasticsearch1", "message": "uncaught exception in thread [main]", 
elasticsearch1    | "stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
elasticsearch1    | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
elasticsearch1    | "at org.elasticsearch.node.Node.<init>(Node.java:615) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "... 6 more",
elasticsearch1    | "Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
elasticsearch1    | "at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
elasticsearch1    | "at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
elasticsearch1    | "at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
elasticsearch1    | "at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]",
elasticsearch1    | "at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]",
elasticsearch1    | "at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]",
elasticsearch1    | "at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]",
elasticsearch1    | uncaught exception in thread [main]
elasticsearch1    | "at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:274) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:211) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:271) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.2.jar:7.6.2]",
elasticsearch1    | "... 6 more"] }
elasticsearch1    | ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];
elasticsearch1    | Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
elasticsearch1    |     at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
elasticsearch1    |     at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
elasticsearch1    |     at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
elasticsearch1    |     at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)
elasticsearch1    |     at java.base/java.nio.file.Files.createDirectory(Files.java:693)
elasticsearch1    |     at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:800)
elasticsearch1    |     at java.base/java.nio.file.Files.createDirectories(Files.java:786)
elasticsearch1    |     at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:274)
elasticsearch1    |     at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:211)
elasticsearch1    |     at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:271)
elasticsearch1    |     at org.elasticsearch.node.Node.<init>(Node.java:277)
elasticsearch1    |     at org.elasticsearch.node.Node.<init>(Node.java:257)
elasticsearch1    |     at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221)
elasticsearch1    |     at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221)
elasticsearch1    |     at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349)
elasticsearch1    |     at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
elasticsearch1    |     at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
elasticsearch1    |     at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
elasticsearch1    |     at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
elasticsearch1    |     at org.elasticsearch.cli.Command.main(Command.java:90)
elasticsearch1    |     at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
elasticsearch1    |     at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
elasticsearch1    | For complete error details, refer to the log at /usr/share/elasticsearch/logs/docker-cluster.log

Solution:

Fix the permissions of the host directory mapped to /usr/share/elasticsearch/data. On my instance that directory is /var/lib/elasticsearch/esdata1.

Run

sudo chown -R 1000:1000 [directory]

e.g.

sudo chown -R 1000:1000 /var/lib/elasticsearch/esdata1

 

Posted by Uli Köhler in ElasticSearch

How to fix landscape-package-reporter: UnicodeDecodeError: ‘utf-8’ codec can’t decode byte

On some servers attached to a landscape instance, I encountered this stacktrace when trying to run sudo landscape-package-reporter:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python3/dist-packages/landscape/client/package/reporter.py", line 92, in <lambda>
    result.addCallback(lambda x: self.request_unknown_hashes())
  File "/usr/lib/python3/dist-packages/landscape/client/package/reporter.py", line 485, in request_unknown_hashes
    self._facade.ensure_channels_reloaded()
  File "/usr/lib/python3/dist-packages/landscape/lib/apt/package/facade.py", line 265, in ensure_channels_reloaded
    self.reload_channels()
  File "/usr/lib/python3/dist-packages/landscape/lib/apt/package/facade.py", line 253, in reload_channels
    version, with_info=False).get_hash()
  File "/usr/lib/python3/dist-packages/landscape/lib/apt/package/facade.py", line 402, in get_package_skeleton
    return build_skeleton_apt(pkg, with_info=with_info, with_unicode=True)
  File "/usr/lib/python3/dist-packages/landscape/lib/apt/package/skeleton.py", line 131, in build_skeleton_apt
    version.record, "Provides", DEB_PROVIDES))
  File "/usr/lib/python3/dist-packages/apt/package.py", line 690, in record
    return Record(self._records.record)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 724: invalid start byte

Tracing down the issue, it was related with a misplaced set of Unicode bytes (EB BF BD) in an old veeam version in version 1.0.0.944 of the veeamsnap package in /var/lib/apt/lists/repository.veeam.com_backup_linux_agent_dpkg_debian_public_dists_stable_veeam_binary-amd64_Packages: The Description field contains this text:

[...] Linux � simple [...]

The strange character is the U+FFFD � REPLACEMENT CHARACTER.

You can fix it by deleting this character. It’s just at the end of /var/lib/apt/lists/repository.veeam.com_backup_linux_agent_dpkg_debian_public_dists_stable_veeam_binary-amd64_Packages. However, if there’s an update for that repository, your change will be overwritten.

In order to fix it (my fix is for landscape-client version 18.01-0ubuntu3.5), I added a try: ... except: ... clause to skeleton.py, which will ignore some properties of the package where the issue occurs:

try:
    relations.update(parse_record_field(
        version.record, "Provides", DEB_PROVIDES))
    relations.add((
        DEB_NAME_PROVIDES,
        "%s = %s" % (version.package.name, version.version)))
    relations.update(parse_record_field(
        version.record, "Pre-Depends", DEB_REQUIRES, DEB_OR_REQUIRES))
    relations.update(parse_record_field(
        version.record, "Depends", DEB_REQUIRES, DEB_OR_REQUIRES))

    relations.add((
        DEB_UPGRADES, "%s < %s" % (version.package.name, version.version)))

    relations.update(parse_record_field(
        version.record, "Conflicts", DEB_CONFLICTS))
    relations.update(parse_record_field(
        version.record, "Breaks", DEB_CONFLICTS))
    skeleton.relations = sorted(relations)

    if with_info:
        skeleton.section = version.section
        skeleton.summary = version.summary
        skeleton.description = version.description
        skeleton.size = version.size
        if version.installed_size > 0:
            skeleton.installed_size = version.installed_size
        if with_unicode and not _PY3:
            skeleton.section = skeleton.section.decode("utf-8")
            skeleton.summary = skeleton.summary.decode("utf-8")
            # Avoid double-decoding package descriptions in build_skeleton_apt,
            # which causes an error with newer python-apt (Xenial onwards)
            if not isinstance(skeleton.description, unicode):
                skeleton.description = skeleton.description.decode("utf-8")
    return skeleton
except UnicodeError:
    return skeleton

Replace /usr/lib/python3/dist-packages/landscape/lib/apt/package/skeleton.py by this:

from landscape.lib.hashlib import sha1

import apt_pkg

from twisted.python.compat import unicode, _PY3


PACKAGE   = 1 << 0
PROVIDES  = 1 << 1
REQUIRES  = 1 << 2
UPGRADES  = 1 << 3
CONFLICTS = 1 << 4

DEB_PACKAGE       = 1 << 16 | PACKAGE
DEB_PROVIDES      = 2 << 16 | PROVIDES
DEB_NAME_PROVIDES = 3 << 16 | PROVIDES
DEB_REQUIRES      = 4 << 16 | REQUIRES
DEB_OR_REQUIRES   = 5 << 16 | REQUIRES
DEB_UPGRADES      = 6 << 16 | UPGRADES
DEB_CONFLICTS     = 7 << 16 | CONFLICTS


class PackageTypeError(Exception):
    """Raised when an unsupported package type is passed to build_skeleton."""


class PackageSkeleton(object):

    section = None
    summary = None
    description = None
    size = None
    installed_size = None
    _hash = None

    def __init__(self, type, name, version):
        self.type = type
        self.name = name
        self.version = version
        self.relations = []

    def add_relation(self, type, info):
        self.relations.append((type, info))

    def get_hash(self):
        """Calculate the package hash.

        If C{set_hash} has been used, that hash will be returned and the
        hash won't be the calculated value.
        """
        if self._hash is not None:
            return self._hash
        # We use ascii here as encoding  for backwards compatibility as it was
        # default encoding for conversion from unicode to bytes in Python 2.7.
        package_info = ("[%d %s %s]" % (self.type, self.name, self.version)
                        ).encode("ascii")
        digest = sha1(package_info)
        self.relations.sort()
        for pair in self.relations:
            digest.update(("[%d %s]" % (pair[0], pair[1])
                           ).encode("ascii"))
        return digest.digest()

    def set_hash(self, package_hash):
        """Set the hash to an explicit value.

        This should be used when the hash is previously known and can't
        be calculated from the relations anymore.

        The only use case for this is package resurrection. We're
        planning on getting rid of package resurrection, and this code
        can be removed when that is done.
        """
        self._hash = package_hash


def relation_to_string(relation_tuple):
    """Convert an apt relation to a string representation.

    @param relation_tuple: A tuple, (name, version, relation). version
        and relation can be the empty string, if the relation is on a
        name only.

    Returns something like "name > 1.0"
    """
    name, version, relation_type = relation_tuple
    relation_string = name
    if relation_type:
        relation_string += " %s %s" % (relation_type, version)
    return relation_string


def parse_record_field(record, record_field, relation_type,
                       or_relation_type=None):
    """Parse an apt C{Record} field and return skeleton relations

    @param record: An C{apt.package.Record} instance with package information.
    @param record_field: The name of the record field to parse.
    @param relation_type: The deb relation that can be passed to
        C{skeleton.add_relation()}
    @param or_relation_type: The deb relation that should be used if
        there is more than one value in a relation.
    """
    relations = set()
    values = apt_pkg.parse_depends(record.get(record_field, ""))
    for value in values:
        value_strings = [relation_to_string(relation) for relation in value]
        value_relation_type = relation_type
        if len(value_strings) > 1:
            value_relation_type = or_relation_type
        relation_string = " | ".join(value_strings)
        relations.add((value_relation_type, relation_string))
    return relations


def build_skeleton_apt(version, with_info=False, with_unicode=False):
    """Build a package skeleton from an apt package.

    @param version: An instance of C{apt.package.Version}
    @param with_info: Whether to extract extra information about the
        package, like description, summary, size.
    @param with_unicode: Whether the C{name} and C{version} of the
        skeleton should be unicode strings.
    """
    name, version_string = version.package.name, version.version
    if with_unicode:
        name, version_string = unicode(name), unicode(version_string)
    skeleton = PackageSkeleton(DEB_PACKAGE, name, version_string)
    relations = set()
    try:
        relations.update(parse_record_field(
            version.record, "Provides", DEB_PROVIDES))
        relations.add((
            DEB_NAME_PROVIDES,
            "%s = %s" % (version.package.name, version.version)))
        relations.update(parse_record_field(
            version.record, "Pre-Depends", DEB_REQUIRES, DEB_OR_REQUIRES))
        relations.update(parse_record_field(
            version.record, "Depends", DEB_REQUIRES, DEB_OR_REQUIRES))

        relations.add((
            DEB_UPGRADES, "%s < %s" % (version.package.name, version.version)))

        relations.update(parse_record_field(
            version.record, "Conflicts", DEB_CONFLICTS))
        relations.update(parse_record_field(
            version.record, "Breaks", DEB_CONFLICTS))
        skeleton.relations = sorted(relations)

        if with_info:
            skeleton.section = version.section
            skeleton.summary = version.summary
            skeleton.description = version.description
            skeleton.size = version.size
            if version.installed_size > 0:
                skeleton.installed_size = version.installed_size
            if with_unicode and not _PY3:
                skeleton.section = skeleton.section.decode("utf-8")
                skeleton.summary = skeleton.summary.decode("utf-8")
                # Avoid double-decoding package descriptions in build_skeleton_apt,
                # which causes an error with newer python-apt (Xenial onwards)
                if not isinstance(skeleton.description, unicode):
                    skeleton.description = skeleton.description.decode("utf-8")
        return skeleton
    except UnicodeError:
        return skeleton

After that, you can run sudo landscape-package-reporter again.

Posted by Uli Köhler in Linux, Python

How to fix Angular 9 @ViewChild Expected 2 arguments, but got 1: An argument for ‘opts’ was not provided.

Problem:

You are trying to compile your Angular 9.x application, but you see an error message like

app/my-component/my-component.component.ts:24:4 - error TS2554: Expected 2 arguments, but got 1.

24   @ViewChild(MyOtherComponent) myOtherComponent: MyOtherComponent;
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  ../node_modules/@angular/core/core.d.ts:7888:47
    7888     (selector: Type<any> | Function | string, opts: {
                                                       ~~~~~~~
    7889         read?: any;
         ~~~~~~~~~~~~~~~~~~~
    7890         static: boolean;
         ~~~~~~~~~~~~~~~~~~~~~~~~
    7891     }): any;
         ~~~~~
    An argument for 'opts' was not provided.

Solution:

Find this line in your code at the location specified in the error message:

@ViewChild(MyOtherComponent) myOtherComponent: MyOtherComponent;

and add

{static: false}

as second argument to the @ViewChild() declaration:

@ViewChild(MyOtherComponent, {static: false}) myOtherComponent: MyOtherComponent;

In most cases, you want to use static: false. See this post on StackOverflow for details on when to use static: true as opposed to static: false.

Posted by Uli Köhler in Angular, Javascript, Typescript