# CMake generated files build/ CMakeCache.txt CMakeFiles/ Makefile cmake_install.cmake install_manifest.txt
# CMake generated files build/ CMakeCache.txt CMakeFiles/ Makefile cmake_install.cmake install_manifest.txt
The following command assumes that you have at least one previous tag in the repository:
git describe --long --tags --dirty
Example output:
v0.1.0-9-g68066f4-dirty
This means that we’re on the 9th commit after the v0.1.0
tag with commit ID g68066f4
and the working tree is dirty, i.e. there are uncommited changes.
The following .gitlab-ci.yml
will build a native executable project using cmake
with a custom docker image:
stages: - build buildmyexe: stage: build image: 'ulikoehler/ubuntu-gcc-cmake:latest' script: - cmake . - make -j4
In this example, we have only one stage – if you have multiple stages, you can specify different images for each of them.
git reset [filename]
will undo a previous git add [filename]
call – one could say, it un-adds filename
Instead of separately running
git submodule init git submodule update
you can just
git submodule update --init
In many cases, you also want to git submodule update
recursively using:
git submodule update --init --recursive
Git does not directly support cloning only a specific file or directory from a repository. However, you can use --depth 1
to clone only one specific revision (as opposed to the entire history) and use --no-checkout
followed by git sparse-checkout set
to checkout not the entire file tree but only a specific file.
In the following example, we’ll checkout only the EMQX config files (in apps/emqx/etc
) from the entire emqx repository:
git clone --depth 1 --branch v5.0.8 --no-checkout https://github.com/emqx/emqx.git cd emqx git sparse-checkout set apps/emqx/etc git checkout v5.0.8
After this command, the emqx
folder will only contain .git
and apps/emqx/etc
.
Note that you can call git sparse-checkout set
multiple times in order to checkout multiple distinct paths.
This example was adapted from this StackOverflow post.
This is useful for cleaning up sensitive data even if you don’t know the specific filename:
git log --pretty=format: --name-only --diff-filter=A | sort -u
Original source: Dustin on StackOverflow
I analyzed a couple of different .gitignore
files for LaTeX floating around the internet.
Clearly, the best one is TeX.gitignore
by Brian Douglas which you can check out here on Github.
# Branch is, for example, "main" export branch=$(git branch --show-current)
Any time I was trying to transfer a project in my docker-hosted gitlab instance. the transfer failed with error 500
and I was presented with the following error log:
==> /var/log/gitlab/gitlab-rails/production.log <== URI::InvalidURIError (query conflicts with opaque): lib/container_registry/client.rb:84:in `repository_tags' app/models/container_repository.rb:94:in `manifest' app/models/container_repository.rb:98:in `tags' app/models/container_repository.rb:118:in `has_tags?' app/models/project.rb:2890:in `has_root_container_repository_tags?' app/models/project.rb:1037:in `has_container_registry_tags?' app/services/projects/transfer_service.rb:61:in `transfer' app/services/projects/transfer_service.rb:35:in `execute' app/controllers/projects_controller.rb:120:in `transfer' app/controllers/application_controller.rb:490:in `set_current_admin' lib/gitlab/session.rb:11:in `with_session' app/controllers/application_controller.rb:481:in `set_session_storage' lib/gitlab/i18n.rb:105:in `with_locale' lib/gitlab/i18n.rb:111:in `with_user_locale' app/controllers/application_controller.rb:475:in `set_locale' app/controllers/application_controller.rb:469:in `set_current_context' lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call' lib/gitlab/middleware/rails_queue_duration.rb:33:in `call' lib/gitlab/middleware/speedscope.rb:13:in `call' lib/gitlab/request_profiler/middleware.rb:17:in `call' lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call' lib/gitlab/metrics/rack_middleware.rb:16:in `block in call' lib/gitlab/metrics/web_transaction.rb:46:in `run' lib/gitlab/metrics/rack_middleware.rb:16:in `call' lib/gitlab/jira/middleware.rb:19:in `call' lib/gitlab/middleware/go.rb:20:in `call' lib/gitlab/etag_caching/middleware.rb:21:in `call' lib/gitlab/middleware/multipart.rb:173:in `call' lib/gitlab/middleware/read_only/controller.rb:50:in `call' lib/gitlab/middleware/read_only.rb:18:in `call' lib/gitlab/middleware/same_site_cookies.rb:27:in `call' lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call' lib/gitlab/middleware/basic_health_check.rb:25:in `call' lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call' lib/gitlab/middleware/request_context.rb:21:in `call' lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call' config/initializers/fix_local_cache_middleware.rb:11:in `call' lib/gitlab/middleware/compressed_json.rb:26:in `call' lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call' lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call' lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call' lib/gitlab/middleware/release_env.rb:13:in `call'
This error seems to occur if you had a docker registry configured in previous gitlab versions (a legacy docker repository) but doesn’t disappear even after deconfiguring the registry.
In order to fix it, I logged into the container using
docker-compose exec gitlab /bin/bash
and edited /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb
using
vi /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb
where on line 2890
you can find the following function:
## # This method is here because of support for legacy container repository # which has exactly the same path like project does, but which might not be # persisted in `container_repositories` table. # def has_root_container_repository_tags? return false unless Gitlab.config.registry.enabled ContainerRepository.build_root_repository(self).has_tags? end
We just want Gitlab to ignore the repository stuff, so we insert return false
after the return false unless Gitlab.config.registry.enabled
line:
## # This method is here because of support for legacy container repository # which has exactly the same path like project does, but which might not be # persisted in `container_repositories` table. # def has_root_container_repository_tags? return false unless Gitlab.config.registry.enabled return false ContainerRepository.build_root_repository(self).has_tags? end
to fake Gitlab into thinking that repository does not have any legacy registries. After that, save the file and
sudo gitlab-ctl restart
after which you should be able to transfer your repositories just fine.
This is the .gitignore
I use for most of my conan
-built executable projects:
CMakeCache.txt CMakeFiles/ Makefile bin/ cmake_install.cmake conan.lock conanbuildinfo.cmake conanbuildinfo.txt conaninfo.txt graph_info.json
It is debatable whether you should include lock files such as conan.lock
. My opinion is that in the development phase they should be ignored but reconsidered when entering into production service.
When running git clone
, you see an error message like
Cloning into 'MyProject'... fatal: unable to access 'https://gitlab.mydomain.com/projects/MyProject.git': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
The quick solution is to prepend
GIT_SSL_NO_VERIFY=true
to the git clone
command, example:
GIT_SSL_NO_VERIFY=true https://gitlab.mydomain.com/projects/MyProject.git
Note that skipping SSL verification is a security risk, so the correct method of fixing this issue is appropriately updating the CA certificates (something like sudo apt install ca-certificates
) but this is sometimes not feasibel since not any outdated computer can be updated easily.
After installing git credential manager core, you see this error message:
fatal: No credential backing store has been selected. Set the GCM_CREDENTIAL_STORE environment variable or the credential.credentialStore Git configuration setting to one of the following options: secretservice : freedesktop.org Secret Service (requires graphical interface) gpg : GNU `pass` compatible credential storage (requires GPG and `pass`) cache : Git's in-memory credential cache plaintext : store credentials in plain-text files (UNSECURE) See https://aka.ms/gcmcore-linuxcredstores for more information.
If you are on a graphical computer, run
git config --global credential.credentialStore secretservice
On a server, if automated i.e. passwordless access is required, use
git config --global credential.credentialStore plaintext
Note that this is insecure since all the passwords are stored in a plaintext file.
This will install git credential manager core on Ubuntu 22.04 or Ubuntu 20.04
wget "https://github.com/GitCredentialManager/git-credential-manager/releases/download/v2.0.886/gcm-linux_amd64.2.0.886.deb" -O /tmp/gcmcore.deb sudo dpkg -i /tmp/gcmcore.deb git-credential-manager configure
I’m currently running 4 separate dockerized gitlab
instances on my server. These tend to consume quite a lot of memory even when not being used for some time.
The gitlab
default is to use 6
unicorn worker processes. By reducing the number of workers to 2
, my gitlab memory consumption decreased by approximately 60%
:
unicorn['worker_processes'] = 2
In my dockerized setup, I justed updated the GITLAB_OMNIBUS_CONFIG
in docker-compose.yml
and restarted the instance. If you didn’t install gitlab
using docker, you might need to sudo gitlab-ctl reconfigure
.
Note that you need at least 2
unicorn workers for gitlab
to work properly. See this issue for details.
Also note that reducing the number of workers to the minimum will likely impact your gitlab
performance in a negative way. Increase the number of workers if you notice a lack in performance.
Most small installation do not need Prometheus, the monitoring tool integrated into Gitlab:
prometheus_monitoring['enable'] = false
sidekiq
is the background job processor integrated into Gitlab. The default concurrency is 25
. I recommend reducing it.
sidekiq['concurrency'] = 2
This might cause background jobs to take longer since they have to wait in queue, but for small installations it does not matter in my experience.
This was recommended on StackOverflow.
postgresql['shared_buffers'] = "256MB"
Setting this too low might cause a heavier IO load and all operations (including website page loads) might be slower.
This is the configuration (combined from all strategies listed above) in order to get down the memory consumption:
# Unicorn config unicorn['worker_processes'] = 2 # PostgreSQL config postgresql['shared_buffers'] = "256MB" # Sidekiq config sidekiq['concurrency'] = 2 # Prometheus config prometheus_monitoring['enable'] = false
Since I had more than 100 repositories in my old gitolite instance and I wanted to migrate to Gitlab a more easy-to-use solution, I developed a
Warning: This is not a finished script but merely a guideline which you need to modify according to your specific needs. I don’t have private repositories in gitolite, so all of my repositories are explicitly listed in the config files. Use on your own responsibility and make a backup!
This does not change or delete any of your repositories in gitolite. Be sure to backup all your repositories in gitolite anyway, just in case!
# Configure git to not ask you for a password every time you are uploading. git config --global credential.helper store # Prepare list of repositories (check the text file and remove invalid names) cat ~/gitolite-admin/*.conf |grep repo | cut -d' ' -f2 > repos.txt # Clone all repos mkdir repos cd repos for i in $(cat ../repos.txt) ; do git clone [email protected]:${i} ; done # Push to Gitlab. This will automatically create a new project as your current user for i in * ; do cd $i && git remote rm origin && git remote add origin "https://gitlab.myserver.org/yourusername/${i}.git" && git push origin master && cd ..; done # Don't forget to make a backup of your gitolite repositories in case anything went wrong!
This script uses the fact that you can directly push to a new repository on Gitlab, creating the project in the process. You don’t need to manually create the project.
While running this script, my Gitlab instance crashed two times while a repository was in the last stage of the git push
process (this tended to happen for small kilobyte-sized repositories) due to heavy swapping induced by heavy memory usage. Restarting gitlab, and re-running the Push to gitlab part of the script fixed this issue.
Note that git config --global credential.helper store
will stay in effect, saving your git
passwords in clear-text. In case you want to restore the default behaviour of keeping them in the RAM for 15 minutes, use git config --global credential.helper cache
after running these commands.
Let’s say you have a git submodule
in MySubmodule
directory. If you have made changes to the repository referred to by MySubmodule
, you might want to update your project to refer to the latest commit.
In order to do this, simply go to the MySubmodule
directory and run git pull
there. You can also git checkout
any other branch.
After that, you need to commit the changes (i.e. which commit git submodule
is referring to currently) in the outer repository.
git status
in the outer repository will show you the changes like this:
Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git checkout -- <file>..." to discard changes in working directory) modified: MySubmodule (new commits)
Use git add MySubmodule
and git commit
normally, or use git commit -a
to commit all staged changes.
You want to run git pull
or some other git
command but you only see this error message:
error: cannot open .git/FETCH_HEAD: Permission denied
This means that the .git directory is not owned by you. The easiest way to fix that is to change the owner of the directory to your user.
First, go to the root directory of the repository using cd
, if you are not already there.
Then,
sudo chown -R $USER: .
In case you don’t have sudo
access on that computer, the easiest way is to copy the repository to a directory where you have write access (e.g. using cp -r
) or even clone the repository again.
Similarly to my previous article about installing Redmine via docker behind a reverse proxy, this article details. Since I am running an instance of Redmine and an instance of Gitlab on the same virtual server, plus tens of other services.
While the Gitlab CE docker container is nicely preconfigured for standalone use on a dedicated VPS, running it behind a reverse proxy is not supported and will lead to a multitude of error messages – in effect, requiring lots of extra work to get up and running.
Note that we will not setup GitLab for SSH access. This is possible using this setup, but usually makes more trouble than it is worth. See this article on how to store git https passwords so you don’t have to enter your password every time.
# Install prerequisites sudo apt-get update sudo apt-get -y install apt-transport-https ca-certificates curl software-properties-common
# Add docker's package signing key curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - # Add repository sudo add-apt-repository -y "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" # Install latest stable docker stable version sudo apt-get update sudo apt-get -y install docker-ce
# Install docker-compose sudo curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose sudo chmod a+x /usr/local/bin/docker-compose # Add current user to the docker group sudo usermod -a -G docker $USER # Enable & start docker service sudo systemctl enable docker sudo systemctl start docker
After running this shell script, log out & login from the system in order for the docker group to be added to the current user.
We will install Gitlab in /var/lib/gitlab
which will host the data directories and the docker-compose
script. You can use any directory if you use it consistently in all the configs (most importantly, docker-compose.yml
and the systemd
service).
# Create directories sudo mkdir /var/lib/gitlab
Next, we’ll create /var/lib/gitlab/docker-compose.yml
.
There’s a couple of things you need to change here:
gitlab_rails['gitlab_email_from']
and gitlab_rails['gitlab_email_display_name']
to whatever sender address & name you want emails to be sent fromgitlab_rails['smtp_address']
, gitlab_rails['smtp_port']
, gitlab_rails['smtp_user_name']
, gitlab_rails['smtp_password']
& gitlab_rails['smtp_domain']
) to a valid SMTP server. In rare cases you also have to change the other gitlab_rails['smtp_...']
settings.gitlab.mydomain.de
to your domain.'9080:80'
means that Gitlab will be mapped to port 9080
on the local PC. This port is chosen somewhat arbitarily – as we will run Gitlab behind an nginx reverse proxy, the port does not need to be any port in particular (as long as you use the same port everywhere), but it may not be used by anything else. You can use any port here, provided that it’s not used for anything else. Leave 80
as-is and only change 9080
if required.gitlab: image: 'gitlab/gitlab-ce:latest' restart: always hostname: 'gitlab.mydomain.de' environment: GITLAB_OMNIBUS_CONFIG: | external_url 'https://gitlab.mydomain.de' letsencrypt['enabled'] = false # Email gitlab_rails['gitlab_email_enabled'] = true gitlab_rails['gitlab_email_from'] = '[email protected]' gitlab_rails['gitlab_email_display_name'] = 'My GitLab' # SMTP gitlab_rails['smtp_enable'] = true gitlab_rails['smtp_address'] = "mail.mydomain.de" gitlab_rails['smtp_port'] = 25 gitlab_rails['smtp_user_name'] = "[email protected]" gitlab_rails['smtp_password'] = "yourSMTPPassword" gitlab_rails['smtp_domain'] = "mydomain.de" gitlab_rails['smtp_authentication'] = "login" gitlab_rails['smtp_enable_starttls_auto'] = true gitlab_rails['smtp_tls'] = true gitlab_rails['smtp_openssl_verify_mode'] = 'none' # Reverse proxy nginx config nginx['listen_port'] = 80 nginx['listen_https'] = false nginx['proxy_set_headers'] = { "X-Forwarded-Proto" => "https", "X-Forwarded-Ssl" => "on", "Host" => "gitlab.mydomain.de", "X-Real-IP" => "$$remote_addr", "X-Forwarded-For" => "$$proxy_add_x_forwarded_for", "Upgrade" => "$$http_upgrade", "Connection" => "$$connection_upgrade" } ports: - '9080:80' volumes: - './config:/etc/gitlab' - './logs:/var/log/gitlab' - './data:/var/opt/gitlab'
Next, we’ll configure the systemd service in /etc/systemd/system/gitlab.service
.
Set User=...
to your preferred user in the [Service]
section. That user needs to be a member of the docker
group. Also check if the WorkingDirectory=...
is correct.
[Unit] Description=Gitlab Requires=docker.service After=docker.service [Service] Restart=always User=root Group=docker WorkingDirectory=/var/lib/gitlab # Shutdown container (if running) when unit is stopped ExecStartPre=/usr/local/bin/docker-compose -f docker-compose.yml down -v # Start container when unit is started ExecStart=/usr/local/bin/docker-compose -f /docker-compose.yml up # Stop container when unit is stopped ExecStop=/usr/local/bin/docker-compose -f docker-compose.yml down -v [Install] WantedBy=multi-user.target
After creating the file, we can enable and start the gitlab service:
sudo systemctl enable gitlab sudo systemctl start gitlab
The output of sudo systemctl start gitlab
should be empty. In case it is
Job for gitlab.service failed because the control process exited with error code. See "systemctl status gitlab.service" and "journalctl -xe" for details.
you can debug the issue using journalctl -xe
and journalctl -e
The first startup usually takes about 10 minutes, so grab at least one cup of coffee. You can follow the progress using journalctl -xefu gitlab
. Once you see lines like
Dec 17 17:28:04 instance-1 docker-compose[4087]: gitlab_1 | {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":28.82,"view":22.82,"db":0.97,"time":"2018-12-17T17:28:03.252Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null}
the startup is finished.
Now you can check if GitLab is running using
wget -O- http://localhost:9080/
(if you changed the port config before, you need to use your custom port in the URL).
If it worked, it will show a debug message output. Since gitlab will automatically redirect you to your domain (gitlab.mydomain.de
in this example) you should see something like
--2018-12-17 17:28:32-- http://localhost:9080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:9080... connected. HTTP request sent, awaiting response... 302 Found Location: https://gitlab.gridbox.de/users/sign_in [following] --2018-12-17 17:28:32-- https://gitlab.mydomain.de/users/sign_in Resolving gitlab.gridbox.de (gitlab.mydomain.de)... 35.198.165.121 Connecting to gitlab.gridbox.de (gitlab.mydomain.de)|35.198.165.121|:443... failed: Connection refused.
Since we have not setup nginx as a reverse proxy yet, it’s totally fine that it’s saying connection refused. The redirection worked if you see the output listed above.
We’ll use nginx to proxy the requests from a certain domain (Using Apache, if you use it already, is also possible but it is outside the scope of this tutorial to tell you how to do that). Install it using
sudo apt -y install nginx
First, you’ll need a domain name with DNS being configured. For this example, we’ll assume that your domain name is gitlab.mydomain.de
! You need to change it to your domain name!
First, we’ll create the config file in /etc/nginx/sites-enabled/gitlab.conf
. Remember to replace gitlab.mydomain.de
by your domain name! If you use a port different from 9080, replace that as ewll.
map $http_upgrade $connection_upgrade { default upgrade; '' close; } server { server_name gitlab.mydomain.de; access_log /var/log/nginx/gitlab.access_log; error_log /var/log/nginx/gitlab.error_log info; location / { proxy_pass http://127.0.0.1:9080; # docker container listens here proxy_read_timeout 3600s; proxy_http_version 1.1; # Websocket connection proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; } listen 80; }
Now run sudo nginx -t
to test if there are any errors in the config file. If everything is alright, you’ll see
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful
Once you have fixed all errors, if any, run sudo service nginx reload
to apply the configuration.
We need to setup a Let’s Encrypt SSL certificate before we can check if Gitlab is working:
First we need to install certbot
and the certbot nginx plugin in order to create & install the certificate in nginx:
sudo apt -y install python3-certbot python3-certbot-nginx
Fortunately certbot automates most of the process of installing & configuring SSL and the certificate. Run
sudo certbot --nginx
It will ask you to enter your Email address and agree to the terms of service and if you want to receive the EFF newsletter.
After that, certbot will ask you to select the correct domain name:
Which names would you like to activate HTTPS for? ------------------------------------------------------------------------------- 1: gitlab.mydomain.de ------------------------------------------------------------------------------- Select the appropriate numbers separated by commas and/or spaces, or leave input blank to select all options shown (Enter 'c' to cancel):
In this case, there is only one domain name (there will be more if you have more domains active on nginx!).
Therefore, enter 1
and press enter. certbot
will now generate the certificate. In case of success you will see an output including a line like
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/gitlab.mydomain.de.conf
Now it will ask you whether to redirect all requests to HTTPS automatically:
Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access. ------------------------------------------------------------------------------- 1: No redirect - Make no further changes to the webserver configuration. 2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for new sites, or if you're confident your site works on HTTPS. You can undo this change by editing your web server's configuration. ------------------------------------------------------------------------------- Select the appropriate number [1-2] then [enter] (press 'c' to cancel):
Choose Redirect
here: Type 2
and press enter. Now you can login to GitLab and finish the installation.
You need to renew the certificate every 3 months for it to stay valid, and run sudo service nginx reload
afterwards to use the new certificate. If you fail to do this, users will see certificate expired error messages and won’t be able to access Gitlab easily! See this post for details on how to mostly automate this process!
Now you can open a browser and have a first look at your new GitLab installation:
Set the new password and then login with the username root and your newly set password.
After that, open the admin area at the top by clicking at the wrench icon in the purple navigation bar at the top.
At the navigation bar at the left, click on Settings (it’s at the bottom – you need to scroll down) and then click on General.
Click the Expand button to the right of Visibility and access controls. Scroll down until you see Enabled Git access protocols and select Only HTTP(S) in the combo box.
Then click the green Save changes button.
Since we have now disabled SSH access (which we didn’t set up in the first place), you can now use GitLab. A good place to start is to create a new project and try checking it out. See this article on how to store git https passwords so you don’t have to enter your git password every time.
Note: If GitLab doesn’t send emails, check config/gitlab.rb
, search for smtp
and if neccessary fix the SMTP settings there. After that, sudo systemctl stop gitlab && sudo systemctl start gitlab
Every time you clone a git repository or push/pull, you have to enter a username and a password (e.g. for GitHub or your GitLab installation).
Instead, you want git to store the password so you only have to enter it once.
Configure the git credential helper to use a plaintext store instead of the default cache:
git config --global credential.helper store
NOTE: This approach will store your passwords in a plaintext file, so depending on your setup this might be a security risk.