Cloud

What image / distribution should you use on Oracle Cloud?

I recommend to always use either CentOS or Ubuntu. This way you will be compatible with other hosting providers and cloud vendors. Additionally, there is much more documentation regarding Ubuntu out there than for distributions like Oracle Linux.

Posted by Uli Köhler in Cloud

Oracle Cloud free tier VM.Standard.E2.1.Micro /proc/cpuinfo

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 23
model           : 1
model name      : AMD EPYC 7551 32-Core Processor
stepping        : 2
microcode       : 0x1000065
cpu MHz         : 1996.246
cache size      : 512 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibpb vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr virt_ssbd arat arch_capabilities
bugs            : sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 3992.49
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 23
model           : 1
model name      : AMD EPYC 7551 32-Core Processor
stepping        : 2
microcode       : 0x1000065
cpu MHz         : 1996.246
cache size      : 512 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibpb vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr virt_ssbd arat arch_capabilities
bugs            : sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 3992.49
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

 

Posted by Uli Köhler in Cloud, Networking

How to disable virtual cloud network firewall on Oracle Cloud

When running VM instances on Oracle Cloud, you might want to use all ports, not just the few ports that are open by default. This post shows how to disable the Virtual Cloud Network firewall altogether. Additionally, you need to configure the instance firewall e.g. via SSH. For Ubuntu, see How to disable instance firewall on Ubuntu on Oracle Cloud.

First login to the cloud network dashboard at https://cloud.oracle.com/networking/vcns

Now click the virtual cloud network:

Now click Security lists on the bottom left:

 

Click the Default security list (which has been automatically created)

Click Add Ingress Rules

Enter source 0.0.0.0/0 (i.e. all IP addresses) and IP protocol All protocols:

Now click Save changes and don’t forget to configure your instance firewall

Posted by Uli Köhler in Cloud, Networking

How to disable instance firewall on Ubuntu on Oracle Cloud

Note: This describes how to disable the firewall on an Ubuntu instance. You additionally need to configure the cloud network security list! See How to disable virtual cloud network firewall on Oracle Cloud for details!

The Oracle firewall is iptables based. We can disable the Ubuntu instance firewall using

sudo iptables -F
sudo netfilter-persistent save

Explanation:

  • iptables -F: Flush (remove all) iptables rules
  • netfilter-persistent save Save empty ruleset to disk so it will be reloaded on reboot.
Posted by Uli Köhler in Cloud, Networking

How to fix Nextcloud 4047 InnoDB refuses to write tables with ROW_FORMAT=COMPRESSED or KEY_BLOCK_SIZE.

Problem:

When trying to maintenance:repair your Nextcloud instance, e.g. during a utf8mb4 upgrade, you see an error message like

ERROR: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 4047 InnoDB refuses to write tables with ROW_FORMAT=COMPRESSED or KEY_BLOCK_SIZE.

Solution:

You need to turn off innodb-read-only-compressed. I do this by starting my MySQL docker with

--skip-innodb-read-only-compressed

Full command line which worked for me:

--transaction-isolation=READ-COMMITTED --binlog-format=ROW --innodb-file-per-table=1 --skip-innodb-read-only-compressed

Full docker-compose.yml nextcloud section:

nextcloud-db:
  image: mariadb
  command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW --innodb-file-per-table=1 --skip-innodb-read-only-compressed
  restart: always
  volumes:
    - ./nextcloud-db:/var/lib/mysql
  environment:
    - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
    - MYSQL_PASSWORD=${MYSQL_PASSWORD}

 

Posted by Uli Köhler in Cloud, Databases, Technologies

Installing apt packages using cloud-init

This cloud-init example installs nginx on Debian- or Ubuntu- based systems:

packages:
- nginx

If you want to enable upgrading packages, use:

package_upgrade: true
packages:
- nginx

 

Posted by Uli Köhler in Cloud, cloud-init

How to fix Google Cloud Build ignoring .dockerignore

Problem:

You want to run a docker image build on Google Cloud Build, but the client is trying to upload a huge context image to Google Cloud even though you have added all your large directories to your .dockerignore and the build works fine locally.

Solution:

Google Cloud Build ignores .dockerignore by design – the equivalent is called .gcloudignore.

You can copy the .dockerignore behaviour for gcloud by running

cp .dockerignore .gcloudignore

 

Posted by Uli Köhler in Cloud, Container, Docker

How to expand Kubernetes Physical Volume Claim (PVC)

Important note: By default, volumes will not be resized immediately but instead require a restart of the associated pod.

First, ensure that you have set allowVolumeExpansion: true for the storage class of your PVC. See our previous post on How to allow Physical Volume Claim (PVC) resize for Kubernetes storage class for more details.

We can expand the volume (named myapp-myapp-pvc-myapp-myapp-1 in this example) by running

kubectl patch pvc/"myapp-myapp-pvc-myapp-myapp-1" \
  --namespace "default" \
  --patch '{"spec": {"resources": {"requests": {"storage": "40Gi"}}}}'

Ensure that you have replaced  the name of the PVC (myapp-myapp-pvc-myapp-myapp-1 in this example) and the storage size. It’s only possible to increase the size of the volume / expand it and not to downsize / shrink it. If your size is less than the previous value, you’ll see this error message:

The PersistentVolumeClaim "myapp-myapp-pvc-myapp-myapp-1" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value

After running this command, the PVC will be in the FileSystemResizePending state.

In order for the update to have effect, you’ll need to force Kubernetes to re-create all the pods for your deployment. To find out how to do this, read our post on How to force restarting all Pods in a Kubernetes Deployment.

For reference, see the official documentation on expanding persistent volumes

Posted by Uli Köhler in Cloud, Kubernetes

How to force restarting all Pods in a Kubernetes Deployment

In contrast to classical deployment managers like systemd or pm2, Kubernetes does not provide a simple restart my application command.

However there’s an easy workaround: If you chance anything in your configuration, even innocuous things that don’t have any effect, Kubernetes will restart your pods.

Consider configuring a rolling update strategy before doing this if you are updating a production application that should have minimal downtime.

In this example we’ll assume you have a StatefulSet your want to update and it’s named elasticsearch-elasticsearch. Be sure to fill in the actual name of your deployment here.

kubectl patch statefulset/elasticsearch-elasticsearch -p \
  "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"dummy-date\":\"`date +'%s'`\"}}}}}"

This will just set a dummy-date annotation which does not have any effect.

You can monitor the update by

kubectl rollout status statefulset/elasticsearch-elasticsearch

Credits for the original solution idea to pstadler on GitHub.

Posted by Uli Köhler in Cloud, Kubernetes

How to configure Google Cloud Kubernetes Elasticsearch Cluster with internal load balancer

Google Cloud offers a convenient way of installing an ElasticSearch cluster on top of a Google Cloud Kubernetes cluster. However, the documentation tells you to expose the ElasticSearch instance using

kubectl patch service/"elasticsearch-elasticsearch-svc" \
  --namespace "default" \
  --patch '{"spec": {"type": "LoadBalancer"}}'

However this command will expost ElasticSearch to an external IP which will make it publically accessible in the default configuration.

Here’s the equivalent command that will expose ElasticSearch to an internal load balancer with an internal IP address that will only be available from Google Cloud.

kubectl patch service/"elasticsearch-elasticsearch-svc" \
  --namespace "default" \
  --patch '{"spec": {"type": "LoadBalancer"}, "metadata": {"annotations": {"cloud.google.com/load-balancer-type": "Internal"}}}'

You might need to replace the name of your service (elasticsearch-elasticsearch-svc in this example) and possibly your namespace.

 

Posted by Uli Köhler in Cloud, ElasticSearch, Kubernetes

How to install MicroK8S (MicroKubernetes) on Ubuntu in 30 seconds

This set of commands will install & start MikroK8S (MikroKubernetes) on Ubuntu and similar Linux distributions.

sudo snap install microk8s --classic
sudo snap install kubectl --classic
sudo microk8s.enable # Autostart on boot
sudo microk8s.start # Start right now
# Wait until microk8s has started
until microk8s.status ; do sleep 1 ; done
# Enable some standard modules
microk8s.enable dashboard registry istio

For reference see the official quickstart manual.

Posted by Uli Köhler in Allgemein, Cloud, Container, Kubernetes

How to fix kubectl unknown shorthand flag: ‘f’ in -f

Problem:

You want to run a Kubernetes kubectl command like

kubectl -f my-app-deployment.yaml

but you see this error message after kubectl prints its entire help page:

unknown shorthand flag: 'f' in -f

Solution:

You are missing an actual command to kubectl. Most likely you want create something on your Kubernetes instance, in which case you want to run this instead:

kubectl create -f my-app-deployment.yaml

You might also want to apply or replace your config instead. Note that apply does not automatically restart your Kubernetes Pods. Read How to fix Kubernetes kubectl apply not restarting pods for more information.

Posted by Uli Köhler in Cloud, Container, Kubernetes

How to rsync to Google Cloud VM instance on command line

If you want to connect to a Google Cloud VM instance (my-instance in this example) from your command line using SSH, use this command:

rsync -Pavz [local file] $(gcloud compute instances list --filter="name=my-instance" --format "get(networkInterfaces[0].accessConfigs[0].natIP)"):

The subcommand (enclosed in $(...) ) finds the correct external IP address for your instance (see How to find IP address of Google Cloud VM instance on command line for more details), so this command boils down to for example

rsync -Pavz [local file] 35.207.77.101:

Using the -Pavz option is not specifically neccessary but these are the options I regularly use for rsync file transfers. You can use any rsync options, Google Cloud does not impose any specific restrictions here. For reference see the rsync manpage.

In case you want to use a different username for the SSH login, you can of course prefix the $(...) section like this:

rsync -Pavz [local file] [email protected]$(gcloud compute instances list --filter="name=my-instance" --format "get(networkInterfaces[0].accessConfigs[0].natIP)"):
Posted by Uli Köhler in Cloud

How to SSH to Google Cloud VM instance on command line

If you want to connect to a Google Cloud VM instance (my-instance in this example) from your command line using SSH, you have two options:

Directly connect using gcloud

This will always work if your instance has SSH enabled, even if it does not have an external IP:

gcloud compute ssh my-instance --zone $(gcloud compute instances list --filter="name=my-instance" --format "get(zone)" | awk -F/ '{print $NF}')

Note that your have to replace my-instance by your actual instance name two times in the command above. The subcommand (enclosed in $(...) ) finds the correct zone for your instance since at the time of writing this article gcloud compute ssh will not work unless you set the correct zone for that instance. See How to find zone of Google Cloud VM instance on command line for more details.

Connect using external IP

You can also use gcloud to get the external IP and connect to it using your standard SSH client.

ssh $(gcloud compute instances list --filter="name=my-instance" --format "get(networkInterfaces[0].accessConfigs[0].natIP)")

This has the added advantage that your will be able to use this in other SSH-like command like rsync.

For reference also see the official manual on Securely Connecting to Instances

Posted by Uli Köhler in Cloud

How to find zone of Google Cloud VM instance on command line

Problem:

You have a VM instance (my-instance in our example) for which you want to find out the zone it’s residing in using the gcloud command line tool.

Solution:

If you just want to see the zone of the instance (remember to replace my-instance by your instance name!), use

gcloud compute instances list --filter="name=my-instance" --format "[box]"

This will format the output nicely and show you more information about your instance. Example output:

┌─────────────┬────────────────┬─────────────────────────────┬─────────────┬─────────────┬───────────────┬─────────┐
│    NAME     │      ZONE      │         MACHINE_TYPE        │ PREEMPTIBLE │ INTERNAL_IP │  EXTERNAL_IP  │  STATUS │
├─────────────┼────────────────┼─────────────────────────────┼─────────────┼─────────────┼───────────────┼─────────┤
│ my-instance │ europe-west3-c │ custom (16 vCPU, 32.00 GiB) │             │ 10.156.0.1  │ 35.207.77.101 │ RUNNING │
└─────────────┴────────────────┴─────────────────────────────┴─────────────┴─────────────┴───────────────┴─────────┘

In this example, the zone is europe-west3-c.

In case you want to see only the zone, use this command instead:

gcloud compute instances list --filter="name=katc-main" --format "get(zone)" | awk -F/ '{print $NF}'

Example output:

europe-west3-c

Also see our other post How to find IP address of Google Cloud VM instance on command line.

In order to see what other information about instances you can see in a similar fashion, use

gcloud compute instances list --filter="name=my-instance" --format "text"
Posted by Uli Köhler in Cloud

How to find IP address of Google Cloud VM instance on command line

Problem:

You have a VM instance (my-instance in our example) for which you want to get the external or internal IP using the gcloud command line tool.

Solution:

If you just want to see the external IP of the instance (remember to replace my-instance by your instance name!), use

gcloud compute instances list --filter="name=my-instance" --format "[box]"

This will format the output nicely and show you more information about your instance. Example output:

┌─────────────┬────────────────┬─────────────────────────────┬─────────────┬─────────────┬───────────────┬─────────┐
│    NAME     │      ZONE      │         MACHINE_TYPE        │ PREEMPTIBLE │ INTERNAL_IP │  EXTERNAL_IP  │  STATUS │
├─────────────┼────────────────┼─────────────────────────────┼─────────────┼─────────────┼───────────────┼─────────┤
│ my-instance │ europe-west3-c │ custom (16 vCPU, 32.00 GiB) │             │ 10.156.0.1  │ 35.207.77.101 │ RUNNING │
└─────────────┴────────────────┴─────────────────────────────┴─────────────┴─────────────┴───────────────┴─────────┘

In this example, the external IP address is 35.207.77.101.

In case you want to see only the IP address, use this command instead:

gcloud compute instances list --filter="name=my-instance" --format "get(networkInterfaces[0].accessConfigs[0].natIP)"

Example output:

35.207.77.101

In order to see only the internal IP address (accessible only from Google Cloud), use

gcloud compute instances list --filter="name=my-instance" --format "get(networkInterfaces[0].networkIP)"

In the linux shell, the result of this command can easily be used as input to other commands. For example, to ping my-instance, use

ping $(gcloud compute instances list --filter="name=katc-main" --format "get(networkInterfaces[0].accessConfigs[0].natIP)")

Also see our related post How to find zone of Google Cloud VM instance on command line

In order to see what other information about instances you can see in a similar fashion, use

gcloud compute instances list --filter="name=my-instance" --format "text"
Posted by Uli Köhler in Cloud

How to fix Kubernetes kubectl apply not restarting pods

Problem:

You made an update to your Kubernetes YAML configuration which you applied with

kubectl apply -f [YAML filename]

but Kubernetes still keeps the old version of the software running.

Solution:

Instead of kubectl apply -f ... use

kubectl replace --force -f [YAML filename]

This will update the configuration on the server and also update the running pods.

Original answer on StackOverflow

Posted by Uli Köhler in Cloud, Container, Kubernetes

How to fix kubectl Unable to connect to the server: dial tcp …:443: i/o timeout

Problem:

You want to create or edit a Kubernetes service but when running e.g.

kubectl create -f my-service.yml

you see an error message similar to this:

Unable to connect to the server: dial tcp 35.198.129.60:443: i/o timeout

Solution:

There are three common reasons for this issue:

  1. Your Kubernetes cluster is not running. Verify that your cluster has been started, e.g. by pinging the IP address.
  2. There are networking issues that prevent you from accessing the cluster. Verify that you can ping the IP and try to track down whether there is a firewall in place preventing the access
  3. You have configured a cluster that does not exist any more.

In case of Google Cloud Kubernetes, case (3) can easily be fixed by configuring Kubernetes to use your current cluster:

gcloud container clusters get-credentials [cluster name] --zone [zone]

This will automatically update the default cluster for kubectl.

In case you don’t know the correct cluster name and zone, use

gcloud container clusters list
Posted by Uli Köhler in Cloud, Container, Kubernetes

How to build & upload a Dockerized application to Google Container Registry in 5 minutes

This post provides an easy example on how to build & upload your application to the private Google Container registry. We assume you have already setup your project and installed Docker. In this example, we’ll build & upload pseudo-perseus v1.0. Since this is a NodeJS-based application, we also assume that you installed a recent version of NodeJS and NPM (see our previous article on how to do that using Ubuntu)

First we configure docker to be able to authenticate to Google:

gcloud auth configure-docker

Now we can checkout the repository and install the NPM packages:

git clone https://github.com/ulikoehler/pseudo-perseus.git
cd pseudo-perseus
git checkout v1.0
npm install

Now we can build the local docker image (we directly name it so that it can be uploaded to the Google Container Registry. Be sure to use the correct google cloud project ID!):

docker build -t eu.gcr.io/myproject-123456/pseudo-perseus:v1.0 .

The next step is to upload the image:

docker push eu.gcr.io/myproject-123456/pseudo-perseus:v1.0

For reference see the official Container Registry documentation.

Posted by Uli Köhler in Cloud, Container, Docker