git

How to ignore “dirty” git submodules

Edit .gitmodules and add ignore = dirty to all modules where you want to ignore the -dirty state.

Example:

[submodule "themes/tailbliss"]
        path = themes/tailbliss
        url = https://github.com/ulikoehler/tailbliss.git
        ignore = dirty

 

Posted by Uli Köhler in git

How to install specific git revision/branch using pip

You can install a specific branch of repository using the pip install git+https://...@branch syntax.

This works with branches and tags:

pip install git+https://github.com/myuser/myrepo.git@mybranch

or, to select a specific revision, just use the revision ID (a shorter revision ID also works):

pip install git+https://github.com/myuser/myrepo.git@aa1edc18e0fb8dc12f550d3b0588eb2c53915097

 

Posted by Uli Köhler in git, Python

How to list all tags of remote git repository using Python

import subprocess

def list_all_tags_for_remote_git_repo(url):
    """
    Given a repository URL, list all tags for that repository
    without cloning it.

    This function use "git ls-remote", so the
    "git" command line program must be available.
    """
    # Run the 'git' command to fetch and list remote tags
    result = subprocess.run([
        "git", "ls-remote", "--tags", repo_url
    ], stdout=subprocess.PIPE, text=True)

    # Process the output to extract tag names
    output_lines = result.stdout.splitlines()
    tags = [
        line.split("refs/tags/")[-1] for line in output_lines
        if "refs/tags/" in line and "^{}" not in line
    ]

    return tags

Usage example:

list_all_tags_for_remote_git_repo("https://github.com/EmbeddedRPC/erpc.git")

Result:

['1.10.0',
 '1.11.0',
 '1.4.0',
 '1.4.1',
 '1.5.0',
 '1.6.0',
 '1.7.0',
 '1.7.1',
 '1.7.2',
 '1.7.3',
 '1.7.4',
 '1.8.0',
 '1.8.1',
 '1.9.0',
 '1.9.1']

 

Posted by Uli Köhler in git, Python

Minimal CMake .gitignore

# CMake generated files
build/
CMakeCache.txt
CMakeFiles/
Makefile
cmake_install.cmake
install_manifest.txt

 

Posted by Uli Köhler in CMake, git

How to generate human-readable description of git commit

The following command assumes that you have at least one previous tag in the repository:

git describe --long --tags --dirty

Example output:

v0.1.0-9-g68066f4-dirty

This means that we’re on the 9th commit after the v0.1.0 tag with commit ID g68066f4 and the working tree is dirty, i.e. there are uncommited changes.

Posted by Uli Köhler in git, Version management

How to specify which docker image to use in .gitlab-ci.yml

The following .gitlab-ci.yml will build a native executable project using cmake with a custom docker image:

stages:
  - build

buildmyexe:
  stage: build
  image: 'ulikoehler/ubuntu-gcc-cmake:latest'
  script:
    - cmake .
    - make -j4

In this example, we have only one stage – if you have multiple stages, you can specify different images for each of them.

Posted by Uli Köhler in Docker, git, GitLab

How to git un-add (undo “git add” command) a file

git reset [filename]

will undo a previous git add [filename] call – one could say, it un-adds filename

Posted by Uli Köhler in git

How to git submodule init & git submodule update in a single command

Instead of separately running

git submodule init
git submodule update

you can just

git submodule update --init

In many cases, you also want to git submodule update recursively using:

git submodule update --init --recursive

 

Posted by Uli Köhler in git

How to git clone only a specific file or directory

Git does not directly support cloning only a specific file or directory from a repository. However, you can use --depth 1 to clone only one specific revision (as opposed to the entire history) and use --no-checkout followed by git sparse-checkout set to checkout not the entire file tree but only a specific file.

In the following example, we’ll checkout only the EMQX config files (in apps/emqx/etc) from the entire emqx repository:

git clone --depth 1 --branch v5.0.8 --no-checkout https://github.com/emqx/emqx.git
cd emqx
git sparse-checkout set apps/emqx/etc
git checkout v5.0.8

After this command, the emqx folder will only contain .git and apps/emqx/etc.

Note that you can call git sparse-checkout set multiple times in order to checkout multiple distinct paths.

This example was adapted from this StackOverflow post.

Posted by Uli Köhler in git, Version management

git: How to list all files that ever existed in the current branch?

This is useful for cleaning up sensitive data even if you don’t know the specific filename:

git log --pretty=format: --name-only --diff-filter=A | sort -u

Original source: Dustin on StackOverflow

Posted by Uli Köhler in git

Where to find a good .gitignore file for LaTeX?

I analyzed a couple of different .gitignore files for LaTeX floating around the internet.

Clearly, the best one is TeX.gitignore by Brian Douglas which you can check out here on Github.

Posted by Uli Köhler in git, Version management

How to use git current branch in bash scripts

# Branch is, for example, "main"
export branch=$(git branch --show-current)
Posted by Uli Köhler in git, Linux, Version management

How I fixed project transfer error Gitlab URI::InvalidURIError (query conflicts with opaque)

Problem:

Any time I was trying to transfer a project in my docker-hosted gitlab instance. the transfer failed with error 500 and I was presented with the following error log:

==> /var/log/gitlab/gitlab-rails/production.log <==
  
URI::InvalidURIError (query conflicts with opaque):
  
lib/container_registry/client.rb:84:in `repository_tags'
app/models/container_repository.rb:94:in `manifest'
app/models/container_repository.rb:98:in `tags'
app/models/container_repository.rb:118:in `has_tags?'
app/models/project.rb:2890:in `has_root_container_repository_tags?'
app/models/project.rb:1037:in `has_container_registry_tags?'
app/services/projects/transfer_service.rb:61:in `transfer'
app/services/projects/transfer_service.rb:35:in `execute'
app/controllers/projects_controller.rb:120:in `transfer'
app/controllers/application_controller.rb:490:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:481:in `set_session_storage'
lib/gitlab/i18n.rb:105:in `with_locale'
lib/gitlab/i18n.rb:111:in `with_user_locale'
app/controllers/application_controller.rb:475:in `set_locale'
app/controllers/application_controller.rb:469:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:26:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

Solution:

This error seems to occur if you had a docker registry configured in previous gitlab versions (a legacy docker repository) but doesn’t disappear even after deconfiguring the registry.

In order to fix it, I logged into the container using

docker-compose exec gitlab /bin/bash

and edited /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb using

vi /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb

where on line 2890 you can find the following function:

##
# This method is here because of support for legacy container repository
# which has exactly the same path like project does, but which might not be
# persisted in `container_repositories` table.
#
def has_root_container_repository_tags?
  return false unless Gitlab.config.registry.enabled

  ContainerRepository.build_root_repository(self).has_tags?
end

We just want Gitlab to ignore the repository stuff, so we insert return false after the return false unless Gitlab.config.registry.enabled line:

##
# This method is here because of support for legacy container repository
# which has exactly the same path like project does, but which might not be
# persisted in `container_repositories` table.
#
def has_root_container_repository_tags?
  return false unless Gitlab.config.registry.enabled
  return false

  ContainerRepository.build_root_repository(self).has_tags?
end

to fake Gitlab into thinking that repository does not have any legacy registries. After that, save the file and

sudo gitlab-ctl restart

after which you should be able to transfer your repositories just fine.

Posted by Uli Köhler in Docker, git

Conan .gitignore for executable projects

This is the .gitignore I use for most of my conan-built executable projects:

CMakeCache.txt
CMakeFiles/
Makefile
bin/
cmake_install.cmake
conan.lock
conanbuildinfo.cmake
conanbuildinfo.txt
conaninfo.txt
graph_info.json

It is debatable whether you should include lock files such as conan.lock. My opinion is that in the development phase they should be ignored but reconsidered when entering into production service.

Posted by Uli Köhler in Conan, git

How to skip SSL certificate verification during git clone

Problem:

When running git clone, you see an error message like

Cloning into 'MyProject'...
fatal: unable to access 'https://gitlab.mydomain.com/projects/MyProject.git': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

Solution:

The quick solution is to prepend

GIT_SSL_NO_VERIFY=true

to the git clone command, example:

GIT_SSL_NO_VERIFY=true https://gitlab.mydomain.com/projects/MyProject.git

Note that skipping SSL verification is a security risk, so the correct method of fixing this issue is appropriately updating the CA certificates (something like sudo apt install ca-certificates) but this is sometimes not feasibel since not any outdated computer can be updated easily.

Posted by Uli Köhler in git, Linux

How to fix git credential manager core: git fatal: No credential backing store has been selected.

Problem:

After installing git credential manager core, you see this error message:

fatal: No credential backing store has been selected.

Set the GCM_CREDENTIAL_STORE environment variable or the credential.credentialStore Git configuration setting to one of the following options:

  secretservice : freedesktop.org Secret Service (requires graphical interface)
  gpg           : GNU `pass` compatible credential storage (requires GPG and `pass`)
  cache         : Git's in-memory credential cache
  plaintext     : store credentials in plain-text files (UNSECURE)

See https://aka.ms/gcmcore-linuxcredstores for more information.

Solution:

If you are on a graphical computer, run

git config --global credential.credentialStore secretservice

On a server, if automated i.e. passwordless access is required, use

git config --global credential.credentialStore plaintext

Note that this is insecure since all the passwords are stored in a plaintext file.

Posted by Uli Köhler in git

How to install git credential manager core on Ubuntu 22.04 / 20.04

This will install git credential manager core on Ubuntu 22.04 or Ubuntu 20.04

wget "https://github.com/git-ecosystem/git-credential-manager/releases/download/v2.5.0/gcm-linux_amd64.2.5.0.deb" -O /tmp/gcmcore.deb
sudo dpkg -i /tmp/gcmcore.deb
git-credential-manager configure
Posted by Uli Köhler in git, Linux

How I reduced gitlab memory consumption in my docker-based setup

I’m currently running 4 separate dockerized gitlab instances on my server. These tend to consume quite a lot of memory even when not being used for some time.

Reduce the number of unicorn worker processes

The gitlab default is to use 6 unicorn worker processes. By reducing the number of workers to 2, my gitlab memory consumption decreased by approximately 60%:

unicorn['worker_processes'] = 2

In my dockerized setup, I justed updated the GITLAB_OMNIBUS_CONFIG in docker-compose.yml and restarted the instance. If you didn’t install gitlab using docker, you might need to sudo gitlab-ctl reconfigure.

Note that you need at least 2 unicorn workers for gitlab to work properly. See this issue for details.

Also note that reducing the number of workers to the minimum will likely impact your gitlab performance in a negative way. Increase the number of workers if you notice a lack in performance.

Disable Prometheus monitoring

Most small installation do not need Prometheus, the monitoring tool integrated into Gitlab:

prometheus_monitoring['enable'] = false

Reduce sidekiq concurrency

sidekiq is the background job processor integrated into Gitlab. The default concurrency is 25. I recommend reducing it.

sidekiq['concurrency'] = 2

This might cause background jobs to take longer since they have to wait in queue, but for small installations it does not matter in my experience.

Reduce the PostgreSQL shared memory

This was recommended on StackOverflow.

postgresql['shared_buffers'] = "256MB"

Setting this too low might cause a heavier IO load and all operations (including website page loads) might be slower.

The complete config

This is the configuration (combined from all strategies listed above) in order to get down the memory consumption:

# Unicorn config
unicorn['worker_processes'] = 2
# PostgreSQL config
postgresql['shared_buffers'] = "256MB"
# Sidekiq config
sidekiq['concurrency'] = 2
# Prometheus config
prometheus_monitoring['enable'] = false

 

Posted by Uli Köhler in Docker, git

How I migrated my gitolite to Gitlab

Since I had more than 100 repositories in my old gitolite instance and I wanted to migrate to Gitlab a more easy-to-use solution, I developed a

Warning: This is not a finished script but merely a guideline which you need to modify according to your specific needs. I don’t have private repositories in gitolite, so all of my repositories are explicitly listed in the config files. Use on your own responsibility and make a backup!

This does not change or delete any of your repositories in gitolite. Be sure to backup all your repositories in gitolite anyway, just in case!

# Configure git to not ask you for a password every time you are uploading.
git config --global credential.helper store
# Prepare list of repositories (check the text file and remove invalid names)
cat ~/gitolite-admin/*.conf |grep repo | cut -d' ' -f2 > repos.txt
# Clone all repos
mkdir repos
cd repos
for i in $(cat ../repos.txt) ; do git clone git@myserver.org:${i} ; done
# Push to Gitlab. This will automatically create a new project as your current user
for i in * ; do cd $i && git remote rm origin && git remote add origin "https://gitlab.myserver.org/yourusername/${i}.git" && git push origin master && cd ..; done 
# Don't forget to make a backup of your gitolite repositories in case anything went wrong!

This script uses the fact that you can directly push to a new repository on Gitlab, creating the project in the process. You don’t need to manually create the project.

While running this script, my Gitlab instance crashed two times while a repository was in the last stage of the git push process (this tended to happen for small kilobyte-sized repositories) due to heavy swapping induced by heavy memory usage. Restarting gitlab, and re-running the Push to gitlab part of the script fixed this issue.

Note that git config --global credential.helper store will stay in effect, saving your git passwords in clear-text. In case you want to restore the default behaviour of keeping them in the RAM for 15 minutes, use git config --global credential.helper cache after running these commands.

Posted by Uli Köhler in git

How to update git submodule to current master or other branch?

Let’s say you have a git submodule in MySubmodule directory. If you have made changes to the repository referred to by MySubmodule, you might want to update your project to refer to the latest commit.

In order to do this, simply go to the MySubmodule directory and run git pull there. You can also git checkout any other branch.

After that, you need to commit the changes (i.e. which commit git submodule is referring to currently) in the outer repository.

git status in the outer repository will show you the changes like this:

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   MySubmodule (new commits)

Use git add MySubmodule and git commit normally, or use git commit -a to commit all staged changes.

Posted by Uli Köhler in git, Version management