How I fixed project transfer error Gitlab URI::InvalidURIError (query conflicts with opaque)


Any time I was trying to transfer a project in my docker-hosted gitlab instance. the transfer failed with error 500 and I was presented with the following error log:

==> /var/log/gitlab/gitlab-rails/production.log <==
URI::InvalidURIError (query conflicts with opaque):
lib/container_registry/client.rb:84:in `repository_tags'
app/models/container_repository.rb:94:in `manifest'
app/models/container_repository.rb:98:in `tags'
app/models/container_repository.rb:118:in `has_tags?'
app/models/project.rb:2890:in `has_root_container_repository_tags?'
app/models/project.rb:1037:in `has_container_registry_tags?'
app/services/projects/transfer_service.rb:61:in `transfer'
app/services/projects/transfer_service.rb:35:in `execute'
app/controllers/projects_controller.rb:120:in `transfer'
app/controllers/application_controller.rb:490:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:481:in `set_session_storage'
lib/gitlab/i18n.rb:105:in `with_locale'
lib/gitlab/i18n.rb:111:in `with_user_locale'
app/controllers/application_controller.rb:475:in `set_locale'
app/controllers/application_controller.rb:469:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:26:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'


This error seems to occur if you had a docker registry configured in previous gitlab versions (a legacy docker repository) but doesn’t disappear even after deconfiguring the registry.

In order to fix it, I logged into the container using

docker-compose exec gitlab /bin/bash

and edited /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb using

vi /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb

where on line 2890 you can find the following function:

# This method is here because of support for legacy container repository
# which has exactly the same path like project does, but which might not be
# persisted in `container_repositories` table.
def has_root_container_repository_tags?
  return false unless Gitlab.config.registry.enabled


We just want Gitlab to ignore the repository stuff, so we insert return false after the return false unless Gitlab.config.registry.enabled line:

# This method is here because of support for legacy container repository
# which has exactly the same path like project does, but which might not be
# persisted in `container_repositories` table.
def has_root_container_repository_tags?
  return false unless Gitlab.config.registry.enabled
  return false


to fake Gitlab into thinking that repository does not have any legacy registries. After that, save the file and

sudo gitlab-ctl restart

after which you should be able to transfer your repositories just fine.

Posted by Uli Köhler in Docker, git

Conan .gitignore for executable projects

This is the .gitignore I use for most of my conan-built executable projects:


It is debatable whether you should include lock files such as conan.lock. My opinion is that in the development phase they should be ignored but reconsidered when entering into production service.

Posted by Uli Köhler in Conan, git

How to skip SSL certificate verification during git clone


When running git clone, you see an error message like

Cloning into 'MyProject'...
fatal: unable to access '': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none


The quick solution is to prepend


to the git clone command, example:


Note that skipping SSL verification is a security risk, so the correct method of fixing this issue is appropriately updating the CA certificates (something like sudo apt install ca-certificates) but this is sometimes not feasibel since not any outdated computer can be updated easily.

Posted by Uli Köhler in git, Linux

How to fix git credential manager core: git fatal: No credential backing store has been selected.


After installing git credential manager core, you see this error message:

fatal: No credential backing store has been selected.

Set the GCM_CREDENTIAL_STORE environment variable or the credential.credentialStore Git configuration setting to one of the following options:

  secretservice : Secret Service (requires graphical interface)
  gpg           : GNU `pass` compatible credential storage (requires GPG and `pass`)
  cache         : Git's in-memory credential cache
  plaintext     : store credentials in plain-text files (UNSECURE)

See for more information.


If you are on a graphical computer, run

git config --global credential.credentialStore secretservice

On a server, if automated i.e. passwordless access is required, use

git config --global credential.credentialStore plaintext

Note that this is insecure since all the passwords are stored in a plaintext file.

Posted by Uli Köhler in git

How I reduced gitlab memory consumption in my docker-based setup

I’m currently running 4 separate dockerized gitlab instances on my server. These tend to consume quite a lot of memory even when not being used for some time.

Reduce the number of unicorn worker processes

The gitlab default is to use 6 unicorn worker processes. By reducing the number of workers to 2, my gitlab memory consumption decreased by approximately 60%:

unicorn['worker_processes'] = 2

In my dockerized setup, I justed updated the GITLAB_OMNIBUS_CONFIG in docker-compose.yml and restarted the instance. If you didn’t install gitlab using docker, you might need to sudo gitlab-ctl reconfigure.

Note that you need at least 2 unicorn workers for gitlab to work properly. See this issue for details.

Also note that reducing the number of workers to the minimum will likely impact your gitlab performance in a negative way. Increase the number of workers if you notice a lack in performance.

Disable Prometheus monitoring

Most small installation do not need Prometheus, the monitoring tool integrated into Gitlab:

prometheus_monitoring['enable'] = false

Reduce sidekiq concurrency

sidekiq is the background job processor integrated into Gitlab. The default concurrency is 25. I recommend reducing it.

sidekiq['concurrency'] = 2

This might cause background jobs to take longer since they have to wait in queue, but for small installations it does not matter in my experience.

Reduce the PostgreSQL shared memory

This was recommended on StackOverflow.

postgresql['shared_buffers'] = "256MB"

Setting this too low might cause a heavier IO load and all operations (including website page loads) might be slower.

The complete config

This is the configuration (combined from all strategies listed above) in order to get down the memory consumption:

# Unicorn config
unicorn['worker_processes'] = 2
# PostgreSQL config
postgresql['shared_buffers'] = "256MB"
# Sidekiq config
sidekiq['concurrency'] = 2
# Prometheus config
prometheus_monitoring['enable'] = false


Posted by Uli Köhler in Docker, git

How I migrated my gitolite to Gitlab

Since I had more than 100 repositories in my old gitolite instance and I wanted to migrate to Gitlab a more easy-to-use solution, I developed a

Warning: This is not a finished script but merely a guideline which you need to modify according to your specific needs. I don’t have private repositories in gitolite, so all of my repositories are explicitly listed in the config files. Use on your own responsibility and make a backup!

This does not change or delete any of your repositories in gitolite. Be sure to backup all your repositories in gitolite anyway, just in case!

# Configure git to not ask you for a password every time you are uploading.
git config --global credential.helper store
# Prepare list of repositories (check the text file and remove invalid names)
cat ~/gitolite-admin/*.conf |grep repo | cut -d' ' -f2 > repos.txt
# Clone all repos
mkdir repos
cd repos
for i in $(cat ../repos.txt) ; do git clone [email protected]:${i} ; done
# Push to Gitlab. This will automatically create a new project as your current user
for i in * ; do cd $i && git remote rm origin && git remote add origin "${i}.git" && git push origin master && cd ..; done 
# Don't forget to make a backup of your gitolite repositories in case anything went wrong!

This script uses the fact that you can directly push to a new repository on Gitlab, creating the project in the process. You don’t need to manually create the project.

While running this script, my Gitlab instance crashed two times while a repository was in the last stage of the git push process (this tended to happen for small kilobyte-sized repositories) due to heavy swapping induced by heavy memory usage. Restarting gitlab, and re-running the Push to gitlab part of the script fixed this issue.

Note that git config --global credential.helper store will stay in effect, saving your git passwords in clear-text. In case you want to restore the default behaviour of keeping them in the RAM for 15 minutes, use git config --global credential.helper cache after running these commands.

Posted by Uli Köhler in git

How to update git submodule to current master or other branch?

Let’s say you have a git submodule in MySubmodule directory. If you have made changes to the repository referred to by MySubmodule, you might want to update your project to refer to the latest commit.

In order to do this, simply go to the MySubmodule directory and run git pull there. You can also git checkout any other branch.

After that, you need to commit the changes (i.e. which commit git submodule is referring to currently) in the outer repository.

git status in the outer repository will show you the changes like this:

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   MySubmodule (new commits)

Use git add MySubmodule and git commit normally, or use git commit -a to commit all staged changes.

Posted by Uli Köhler in git, Version management

How to fix git ‘error: cannot open .git/FETCH_HEAD: Permission denied’


You want to run git pull or some other git command but you only see this error message:

error: cannot open .git/FETCH_HEAD: Permission denied


This means that the .git directory is not owned by you. The easiest way to fix that is to change the owner of the directory to your user.

First, go to the root directory of the repository using cd, if you are not already there.


sudo chown -R $USER: .

In case you don’t have sudo access on that computer, the easiest way is to copy the repository to a directory where you have write access (e.g. using cp -r) or even clone the repository again.

Posted by Uli Köhler in git, Version management

Running Gitlab CE via docker behind a reverse proxy on Ubuntu

Similarly to my previous article about installing Redmine via docker behind a reverse proxy, this article details. Since I am running an instance of Redmine and an instance of Gitlab on the same virtual server, plus tens of other services.

While the Gitlab CE docker container is nicely preconfigured for standalone use on a dedicated VPS, running it behind a reverse proxy is not supported and will lead to a multitude of error messages – in effect, requiring lots of extra work to get up and running.

Note that we will not setup GitLab for SSH access. This is possible using this setup, but usually makes more trouble than it is worth. See this article on how to store git https passwords so you don’t have to enter your password every time.

Installing Docker & Docker-Compose

# Install prerequisites
sudo apt-get update
sudo apt-get -y install apt-transport-https ca-certificates curl software-properties-common
# Add docker's package signing key
curl -fsSL | sudo apt-key add -
# Add repository
sudo add-apt-repository -y "deb [arch=amd64] $(lsb_release -cs) stable"
# Install latest stable docker stable version
sudo apt-get update
sudo apt-get -y install docker-ce
# Install docker-compose
sudo curl -L "$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod a+x /usr/local/bin/docker-compose

# Add current user to the docker group
sudo usermod -a -G docker $USER
# Enable & start docker service
sudo systemctl enable docker
sudo systemctl start docker

After running this shell script, log out & login from the system in order for the docker group to be added to the current user.

Creating the directory & docker-compose configuration

We will install Gitlab in /var/lib/gitlab which will host the data directories and the docker-compose script. You can use any directory if you use it consistently in all the configs (most importantly, docker-compose.yml and the systemd service).

# Create directories
sudo mkdir /var/lib/gitlab

Next, we’ll create /var/lib/gitlab/docker-compose.yml.

There’s a couple of things you need to change here:

  • Set gitlab_rails['gitlab_email_from'] and gitlab_rails['gitlab_email_display_name'] to whatever sender address & name you want emails to be sent from
  • Set the SMTP credentials (gitlab_rails['smtp_address'], gitlab_rails['smtp_port'], gitlab_rails['smtp_user_name'], gitlab_rails['smtp_password'] & gitlab_rails['smtp_domain']) to a valid SMTP server. In rare cases you also have to change the other gitlab_rails['smtp_...'] settings.
  • You need to change every 4 occurrences of to your domain.
  • The ports configuration, in this case '9080:80' means that Gitlab will be mapped to port 9080 on the local PC. This port is chosen somewhat arbitarily – as we will run Gitlab behind an nginx reverse proxy, the port does not need to be any port in particular (as long as you use the same port everywhere), but it may not be used by anything else. You can use any port here, provided that it’s not used for anything else. Leave 80 as-is and only change 9080 if required.
   image: 'gitlab/gitlab-ce:latest'
   restart: always
   hostname: ''
       external_url ''
       letsencrypt['enabled'] = false
       # Email
       gitlab_rails['gitlab_email_enabled'] = true
       gitlab_rails['gitlab_email_from'] = '[email protected]'
       gitlab_rails['gitlab_email_display_name'] = 'My GitLab'
       # SMTP
       gitlab_rails['smtp_enable'] = true
       gitlab_rails['smtp_address'] = ""
       gitlab_rails['smtp_port'] = 25
       gitlab_rails['smtp_user_name'] = "[email protected]"
       gitlab_rails['smtp_password'] = "yourSMTPPassword"
       gitlab_rails['smtp_domain'] = ""
       gitlab_rails['smtp_authentication'] = "login"
       gitlab_rails['smtp_enable_starttls_auto'] = true
       gitlab_rails['smtp_tls'] = true
       gitlab_rails['smtp_openssl_verify_mode'] = 'none'
       # Reverse proxy nginx config
       nginx['listen_port'] = 80
       nginx['listen_https'] = false
       nginx['proxy_set_headers'] = {
         "X-Forwarded-Proto" => "https",
         "X-Forwarded-Ssl" => "on",
         "Host" => "",
         "X-Real-IP" => "$$remote_addr",
         "X-Forwarded-For" => "$$proxy_add_x_forwarded_for",
         "Upgrade" => "$$http_upgrade",
         "Connection" => "$$connection_upgrade"
     - '9080:80'
     - './config:/etc/gitlab'
     - './logs:/var/log/gitlab'
     - './data:/var/opt/gitlab'

Setting up the systemd service

Next, we’ll configure the systemd service in /etc/systemd/system/gitlab.service.

Set User=... to your preferred user in the [Service] section. That user needs to be a member of the docker group. Also check if the WorkingDirectory=... is correct.


# Shutdown container (if running) when unit is stopped
ExecStartPre=/usr/local/bin/docker-compose -f docker-compose.yml down -v
# Start container when unit is started
ExecStart=/usr/local/bin/docker-compose -f /docker-compose.yml up
# Stop container when unit is stopped
ExecStop=/usr/local/bin/docker-compose -f docker-compose.yml down -v


After creating the file, we can enable and start the gitlab service:

sudo systemctl enable gitlab
sudo systemctl start gitlab

The output of sudo systemctl start gitlab should be empty. In case it is

Job for gitlab.service failed because the control process exited with error code.
See "systemctl status gitlab.service" and "journalctl -xe" for details.

you can debug the issue using journalctl -xe and journalctl -e

The first startup usually takes about 10 minutes, so grab at least one cup of coffee. You can follow the progress using journalctl -xefu gitlab. Once you see lines like

Dec 17 17:28:04 instance-1 docker-compose[4087]: gitlab_1  | {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":28.82,"view":22.82,"db":0.97,"time":"2018-12-17T17:28:03.252Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null}

the startup is finished.

Now you can check if GitLab is running using

wget -O- http://localhost:9080/

(if you changed the port config before, you need to use your custom port in the URL).

If it worked, it will show a debug message output. Since gitlab will automatically redirect you to your domain ( in this example) you should see something like

--2018-12-17 17:28:32--  http://localhost:9080/
Resolving localhost (localhost)...
Connecting to localhost (localhost)||:9080... connected.
HTTP request sent, awaiting response... 302 Found
Location: [following]
--2018-12-17 17:28:32--
Resolving (
Connecting to (||:443... failed: Connection refused.

Since we have not setup nginx as a reverse proxy yet, it’s totally fine that it’s saying connection refused. The redirection worked if you see the output listed above.

Setting up the nginx reverse proxy (optional but recommended)

We’ll use nginx to proxy the requests from a certain domain (Using Apache, if you use it already, is also possible but it is outside the scope of this tutorial to tell you how to do that). Install it using

sudo apt -y install nginx

First, you’ll need a domain name with DNS being configured. For this example, we’ll assume that your domain name is ! You need to change it to your domain name!

First, we’ll create the config file in /etc/nginx/sites-enabled/gitlab.conf. Remember to replace by your domain name! If you use a port different from 9080, replace that as ewll.

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;

server {

    access_log /var/log/nginx/gitlab.access_log;
    error_log /var/log/nginx/gitlab.error_log info;

    location / {
        proxy_pass; # docker container listens here
        proxy_read_timeout 3600s;
        proxy_http_version 1.1;
        # Websocket connection
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

    listen 80;

Now run sudo nginx -t to test if there are any errors in the config file. If everything is alright, you’ll see

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Once you have fixed all errors, if any, run sudo service nginx reload to apply the configuration.

We need to setup a Let’s Encrypt SSL certificate before we can check if Gitlab is working:

Securing the nginx reverse proxy using Let’s Encrypt

First we need to install certbot and the certbot nginx plugin in order to create & install the certificate in nginx:

sudo apt -y install python3-certbot python3-certbot-nginx

Fortunately certbot automates most of the process of installing & configuring SSL and the certificate. Run

sudo certbot --nginx

It will ask you to enter your Email address and agree to the terms of service and if you want to receive the EFF newsletter.

After that, certbot will ask you to select the correct domain name:

Which names would you like to activate HTTPS for?
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel):

In this case, there is only one domain name (there will be more if you have more domains active on nginx!).

Therefore, enter 1 and press enter. certbot will now generate the certificate. In case of success you will see an output including a line like

Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/

Now it will ask you whether to redirect all requests to HTTPS automatically:

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 

Choose Redirect here: Type 2 and press enter. Now you can login to GitLab and finish the installation.

You need to renew the certificate every 3 months for it to stay valid, and run sudo service nginx reload afterwards to use the new certificate. If you fail to do this, users will see certificate expired error messages and won’t be able to access Gitlab easily! See this post for details on how to mostly automate this process!

Setting up Gitlab

Now you can open a browser and have a first look at your new GitLab installation:

Set the new password and then login with the username root and your newly set password.

After that, open the admin area at the top by clicking at the wrench icon in the purple navigation bar at the top.

At the navigation bar at the left, click on Settings (it’s at the bottom – you need to scroll down) and then click on General.

Click the Expand button to the right of Visibility and access controls. Scroll down until you see Enabled Git access protocols and select Only HTTP(S) in the combo box.

Then click the green Save changes button.

Since we have now disabled SSH access (which we didn’t set up in the first place), you can now use GitLab. A good place to start is to create a new project and try checking it out. See this article on how to store git https passwords so you don’t have to enter your git password every time.

Note: If GitLab doesn’t send emails, check config/gitlab.rb, search for smtp and if neccessary fix the SMTP settings there. After that, sudo systemctl stop gitlab && sudo systemctl start gitlab

Posted by Uli Köhler in Container, Docker, git, nginx, Version management

How to store git HTTPS passwords / credentials permanently so you only have to enter them once

Note: Since 2021, I strongly recommend to use git credential manager core instead of using this older method. See How to install git credential manager core on Ubuntu 20.04


Every time you clone a git repository or push/pull, you have to enter a username and a password (e.g. for GitHub or your GitLab installation).
Instead, you want git to store the password so you only have to enter it once.


Configure the git credential helper to use a plaintext store instead of the default cache:

git config --global credential.helper store

NOTE: This approach will store your passwords in a plaintext file, so depending on your setup this might be a security risk.

Posted by Uli Köhler in git, Version management

How to solve git: fatal: No configured push destination


You  have initialized a git repository in a folder using

git init

Now that you have made some commits, you want to use

git push

but you get the following error message:

fatal: No configured push destination.
Either specify the URL from the command-line or configure a remote repository using

    git remote add <name> <url>

and then push using the remote name

    git push <name>


As you initialized your repository using git init, git does not know which server to contact when you use git push.

Therefore, we’ll have to add a server (called remote in git terminology) to the repository:

git remote add origin [email protected]:yourusername/yourrepository.git

Remember to replace [email protected]:yourusername/yourrepository.git with the correct URL for your repository. Valid example URLs include:

  • [email protected]:ulikoehler/UliEngineering.git

This adds a server (remote add) named origin with the URL [email protected]:yourusername/yourrepository.git.

The URL (last argument) depends on the server you use, for GitHub, you can get the URL (HTTPS or SSH, both will work) by clicking the green Clone or Download button.

Now you can push your existing data to the server. git push by itself won’t work for the first time, because git doesn’t know automatically that you want to push to origin. Therefore we have to tell it using --set-upstream that future git push commands shall automatically push to origin:

git push --set-upstream origin master

If this command lists an error, you likely used the wrong URL for the repository or you don’t use the correct credentials (username/password, SSH key etc).

From now on, you can just use

git push

every time you’ve made a commit in order to push it to the server.

Note: origin is no special name, it’s just the name git uses for the server when you git clone a repository. Therefore it’s the standard name for your main server to push to. Similarly, git uses master as the default branch name.

Posted by Uli Köhler in git, Version management

git svn: Clone latest revision only


You want to use git-svn to clone a SVN repository, but you don’t want to clone the entire history (which can be quite slow) but only the latest revision.

Continue reading →

Posted by Uli Köhler in git, Shell, Subversion, Version management