Project management

How I fixed project transfer error Gitlab URI::InvalidURIError (query conflicts with opaque)

Problem:

Any time I was trying to transfer a project in my docker-hosted gitlab instance. the transfer failed with error 500 and I was presented with the following error log:

==> /var/log/gitlab/gitlab-rails/production.log <==
  
URI::InvalidURIError (query conflicts with opaque):
  
lib/container_registry/client.rb:84:in `repository_tags'
app/models/container_repository.rb:94:in `manifest'
app/models/container_repository.rb:98:in `tags'
app/models/container_repository.rb:118:in `has_tags?'
app/models/project.rb:2890:in `has_root_container_repository_tags?'
app/models/project.rb:1037:in `has_container_registry_tags?'
app/services/projects/transfer_service.rb:61:in `transfer'
app/services/projects/transfer_service.rb:35:in `execute'
app/controllers/projects_controller.rb:120:in `transfer'
app/controllers/application_controller.rb:490:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:481:in `set_session_storage'
lib/gitlab/i18n.rb:105:in `with_locale'
lib/gitlab/i18n.rb:111:in `with_user_locale'
app/controllers/application_controller.rb:475:in `set_locale'
app/controllers/application_controller.rb:469:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:26:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

Solution:

This error seems to occur if you had a docker registry configured in previous gitlab versions (a legacy docker repository) but doesn’t disappear even after deconfiguring the registry.

In order to fix it, I logged into the container using

docker-compose exec gitlab /bin/bash

and edited /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb using

vi /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb

where on line 2890 you can find the following function:

##
# This method is here because of support for legacy container repository
# which has exactly the same path like project does, but which might not be
# persisted in `container_repositories` table.
#
def has_root_container_repository_tags?
  return false unless Gitlab.config.registry.enabled

  ContainerRepository.build_root_repository(self).has_tags?
end

We just want Gitlab to ignore the repository stuff, so we insert return false after the return false unless Gitlab.config.registry.enabled line:

##
# This method is here because of support for legacy container repository
# which has exactly the same path like project does, but which might not be
# persisted in `container_repositories` table.
#
def has_root_container_repository_tags?
  return false unless Gitlab.config.registry.enabled
  return false

  ContainerRepository.build_root_repository(self).has_tags?
end

to fake Gitlab into thinking that repository does not have any legacy registries. After that, save the file and

sudo gitlab-ctl restart

after which you should be able to transfer your repositories just fine.

Posted by Uli Köhler in Docker, git

Conan .gitignore for executable projects

This is the .gitignore I use for most of my conan-built executable projects:

CMakeCache.txt
CMakeFiles/
Makefile
bin/
cmake_install.cmake
conan.lock
conanbuildinfo.cmake
conanbuildinfo.txt
conaninfo.txt
graph_info.json

It is debatable whether you should include lock files such as conan.lock. My opinion is that in the development phase they should be ignored but reconsidered when entering into production service.

Posted by Uli Köhler in Conan, git

How to initialize new C++ executable project using conan

In the directory where you want to create the project, run (obviously, replace mypackageby the name of everywhere in our code!)

conan new mypackage/1.0.0 -s

-s is important here. It tells conan that the source is available in the project directory and it should not be pulled using git etc.

This will initialize a shared library project, which we will now modify to build an executable.

First, open src/CMakeLists.txt and change add_library() to add_executable(), for example

add_library(mypackage mypackage.cpp)

to

add_executable(mypackage mypackage.cpp)

Now we shall modify conanfile.py to properly build & install the executable:

    def package(self):
        self.copy("*.h", dst="include", src="src")
        self.copy("*.lib", dst="lib", keep_path=False)
        self.copy("*.dll", dst="bin", keep_path=False)
        self.copy("*.dylib*", dst="lib", keep_path=False)
        self.copy("*.so", dst="lib", keep_path=False)
        self.copy("*.a", dst="lib", keep_path=False)
    def package_info(self):
        self.cpp_info.libs = ["NoxecoDB"]

with

    def package(self):
        self.copy("mypackage", src="bin", dst="bin", keep_path=False)

    def package_info(self):
        self.env_info.PATH = os.path.join(self.package_folder, "bin")

    def deploy(self):
        self.copy("mypackage", dst="bin", src="bin")

and add

import os.path

to the top of the file

Now let’s add a main() function to  src/mypackage.cpp. Replace the file’s content with

#include <iostream>

int main(int argc, char** argv) {
    std::cout << "Hello World!" <<std::endl;
}

Now it’s time to install the dependencies & build the project using

conan install . && conan build .

The executable will be in

bin/mypackage

You can run it using

./bin/mypackage

which will print

Hello World!

I also recommend to add this .gitignore:

CMakeCache.txt
CMakeFiles/
Makefile
bin/
cmake_install.cmake
conan.lock
conanbuildinfo.cmake
conanbuildinfo.txt
conaninfo.txt
graph_info.json

 

Posted by Uli Köhler in C/C++, Conan

How to skip SSL certificate verification during git clone

Problem:

When running git clone, you see an error message like

Cloning into 'MyProject'...
fatal: unable to access 'https://gitlab.mydomain.com/projects/MyProject.git': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

Solution:

The quick solution is to prepend

GIT_SSL_NO_VERIFY=true

to the git clone command, example:

GIT_SSL_NO_VERIFY=true https://gitlab.mydomain.com/projects/MyProject.git

Note that skipping SSL verification is a security risk, so the correct method of fixing this issue is appropriately updating the CA certificates (something like sudo apt install ca-certificates) but this is sometimes not feasibel since not any outdated computer can be updated easily.

Posted by Uli Köhler in git, Linux

How to fix conan install ERROR: Missing binary

Problem:

When running

conan install .

you see  ERROR: Missing binary error messages like this:

Installing (downloading, building) binaries...
ERROR: Missing binary: cpr/1.6.2:76fe60a9d5c829a2384b0f578695b5a6357b8acc
ERROR: Missing binary: gtest/cci.20210126:c37dc23725d84483088a68c29e2dd7122c328359
ERROR: Missing binary: jsoncpp/1.9.4:718278bac9f92b77a9a44bfbe1aa00d1c8344d51
ERROR: Missing binary: libcurl/7.78.0:d287dc966931230205222ceabf47400733e72fa3
ERROR: Missing binary: openssl/1.1.1l:c3baf9fae083edda2e0c5ef3337b6a111016a898
ERROR: Missing binary: zlib/1.2.11:c3baf9fae083edda2e0c5ef3337b6a111016a898

Solution:

Use the --build=missing flag for conan install to enable building the dependencies for which no binaries are available:

conan install . --build=missing

Complete error log example

Configuration:
[settings]
arch=armv8
arch_build=armv8
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=9
os=Linux
os_build=Linux
[options]
[build_requires]
[env]

openssl/1.1.1l: Not found in local cache, looking in remotes...
openssl/1.1.1l: Trying with 'conancenter'...
Downloading conanmanifest.txt completed [0.26k]                                          
Downloading conanfile.py completed [40.47k]                                              
Downloading conan_export.tgz completed [0.25k]                                           
Decompressing conan_export.tgz completed [0.00k]                                         
openssl/1.1.1l: Downloaded recipe revision 0
cpr/1.6.2: Not found in local cache, looking in remotes...
cpr/1.6.2: Trying with 'conancenter'...
Downloading conanmanifest.txt completed [0.25k]                                          
Downloading conanfile.py completed [9.62k]                                               
Downloading conan_export.tgz completed [0.30k]                                           
Decompressing conan_export.tgz completed [0.00k]                                         
cpr/1.6.2: Downloaded recipe revision 0
WARN: cpr/1.6.2: requirement libcurl/7.80.0 overridden by elasticlient/0.2 to libcurl/7.78.0 
libcurl/7.78.0: Not found in local cache, looking in remotes...
libcurl/7.78.0: Trying with 'conancenter'...
Downloading conanmanifest.txt completed [0.70k]                                          
Downloading conanfile.py completed [27.11k]                                              
Downloading conan_export.tgz completed [0.23k]                                           
Decompressing conan_export.tgz completed [0.00k]                                         
libcurl/7.78.0: Downloaded recipe revision 0
jsoncpp/1.9.4: Not found in local cache, looking in remotes...
jsoncpp/1.9.4: Trying with 'conancenter'...
Downloading conanmanifest.txt completed [0.65k]                                          
Downloading conanfile.py completed [4.90k]                                               
Downloading conan_export.tgz completed [0.25k]                                           
Decompressing conan_export.tgz completed [0.00k]                                         
jsoncpp/1.9.4: Downloaded recipe revision 0
gtest/cci.20210126: Not found in local cache, looking in remotes...
gtest/cci.20210126: Trying with 'conancenter'...
Downloading conanmanifest.txt completed [0.39k]                                          
Downloading conanfile.py completed [8.14k]                                               
Downloading conan_export.tgz completed [0.32k]                                           
Decompressing conan_export.tgz completed [0.00k]                                         
gtest/cci.20210126: Downloaded recipe revision 0
conanfile.py (elasticlient/0.2): Installing package
Requirements
    cpr/1.6.2 from 'conancenter' - Downloaded
    jsoncpp/1.9.4 from 'conancenter' - Downloaded
    libcurl/7.78.0 from 'conancenter' - Downloaded
    openssl/1.1.1l from 'conancenter' - Downloaded
    zlib/1.2.11 from 'conancenter' - Cache
Packages
    cpr/1.6.2:76fe60a9d5c829a2384b0f578695b5a6357b8acc - Missing
    jsoncpp/1.9.4:718278bac9f92b77a9a44bfbe1aa00d1c8344d51 - Missing
    libcurl/7.78.0:d287dc966931230205222ceabf47400733e72fa3 - Missing
    openssl/1.1.1l:c3baf9fae083edda2e0c5ef3337b6a111016a898 - Missing
    zlib/1.2.11:c3baf9fae083edda2e0c5ef3337b6a111016a898 - Missing
Build requirements
    gtest/cci.20210126 from 'conancenter' - Downloaded
Build requirements packages
    gtest/cci.20210126:c37dc23725d84483088a68c29e2dd7122c328359 - Missing

Installing (downloading, building) binaries...
ERROR: Missing binary: cpr/1.6.2:76fe60a9d5c829a2384b0f578695b5a6357b8acc
ERROR: Missing binary: gtest/cci.20210126:c37dc23725d84483088a68c29e2dd7122c328359
ERROR: Missing binary: jsoncpp/1.9.4:718278bac9f92b77a9a44bfbe1aa00d1c8344d51
ERROR: Missing binary: libcurl/7.78.0:d287dc966931230205222ceabf47400733e72fa3
ERROR: Missing binary: openssl/1.1.1l:c3baf9fae083edda2e0c5ef3337b6a111016a898
ERROR: Missing binary: zlib/1.2.11:c3baf9fae083edda2e0c5ef3337b6a111016a898

 

Posted by Uli Köhler in Conan

How to fix git credential manager core: git fatal: No credential backing store has been selected.

Problem:

After installing git credential manager core, you see this error message:

fatal: No credential backing store has been selected.

Set the GCM_CREDENTIAL_STORE environment variable or the credential.credentialStore Git configuration setting to one of the following options:

  secretservice : freedesktop.org Secret Service (requires graphical interface)
  gpg           : GNU `pass` compatible credential storage (requires GPG and `pass`)
  cache         : Git's in-memory credential cache
  plaintext     : store credentials in plain-text files (UNSECURE)

See https://aka.ms/gcmcore-linuxcredstores for more information.

Solution:

If you are on a graphical computer, run

git config --global credential.credentialStore secretservice

On a server, if automated i.e. passwordless access is required, use

git config --global credential.credentialStore plaintext

Note that this is insecure since all the passwords are stored in a plaintext file.

Posted by Uli Köhler in git

How to create conan debug profile

Run this command to create a debug profile from your default profile:

cp ~/.conan/profiles/default ~/.conan/profiles/debug && sed -i -e 's/Release/Debug/g' ~/.conan/profiles/debug

Example output from conan profile show debug:

[settings]
os=Linux
os_build=Linux
arch=x86_64
arch_build=x86_64
compiler=gcc
compiler.version=9
compiler.libcxx=libstdc++
build_type=Debug
[options]
[conf]
[build_requires]
[env]

 

Posted by Uli Köhler in C/C++, Conan

How to list local conan packages

In order to list local conan packages, run

conan search

Example output:

$ conan search
Existing package recipes:

cpr/1.6.2
elasticlient/0.2
elasticlient/[email protected]/testing
gtest/cci.20210126
jsoncpp/1.9.4
libcurl/7.69.1
libcurl/7.78.0
openssl/1.1.1l
zlib/1.2.11

 

Posted by Uli Köhler in Conan

How to fix Conan CMake “Could NOT find OpenSSL”

Problem:

When building your conan package, you see this CMake error:

CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
  system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY
  OPENSSL_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindOpenSSL.cmake:447 (find_package_handle_standard_args)
  ../_deps/curl-src/CMakeLists.txt:365 (find_package)

even though your conanfile.py declares OpenSSL as a dependency:

class MyPackageConan(ConanFile):
    # ...
    requires = ("openssl/1.1.1l", )

Solution:

Make sure that the project’s CMakeLists.txt contains these lines:

include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
conan_basic_setup()

Typically, the way to do this is to use this code which is automatically generated by conan new:

        # This small hack might be useful to guarantee proper /MT /MD linkage
        # in MSVC if the packaged project doesn't have variables to set it
        # properly
        tools.replace_in_file("hello/CMakeLists.txt", "PROJECT(HelloWorld)",
                              '''PROJECT(HelloWorld)
include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
conan_basic_setup()''')

in source() like this:

    def source(self):
        self.run("git clone https://github.com/conan-io/hello.git")
        # This small hack might be useful to guarantee proper /MT /MD linkage
        # in MSVC if the packaged project doesn't have variables to set it
        # properly
        tools.replace_in_file("hello/CMakeLists.txt", "PROJECT(HelloWorld)",
                              '''PROJECT(HelloWorld)
include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
conan_basic_setup()''')

Note that you need to replace both hello/CMakeLists.txt with the correct filename (typically, just replace hello by). Also, you need to replace both instances PROJECT(HelloWorld) by the actual line from your CMakeLists.txt in order for the replace command to work.

Posted by Uli Köhler in C/C++, CMake, Conan

How to add build-only dependencies in Conan recipe

Just add build_requires = ('dep1/version1', 'dep1/version2') to your conanfile.py:

class MyPackageConan(ConanFile):
    # ...
    requires = ("openssl/1.1.1l", )
    build_requires = ("jsoncpp/1.9.4", )

 

Posted by Uli Köhler in C/C++, Conan

How to run ‘git submodule update’ in Conan recipe

If you want to run git submodule update in your conanfile.py, you can simple add a self.run() command after the git clone command:

self.run("git clone https://github.com/conan-io/hello.git")
self.run("cd hello && git submodule update --init --recursive")

Full example:

self.run("git clone https://github.com/seznam/elasticlient.git -b version-0.2")
self.run("cd elasticlient && git submodule update --init --recursive")

Note that you need to cd <directory> && git submodule update

Posted by Uli Köhler in C/C++, Conan

How to add conan CMake build definition

When building a library using conan, this is the default conanfile.py definition for building using CMake

cmake = CMake(self)
cmake.configure(source_folder="hello")
cmake.build()

You can easily add definitions like this:

cmake.definitions["MY_FLAG"] = "1"

This will generate -DMY_FLAG=1 when calling CMake

Full example:

cmake = CMake(self)
cmake.definitions["USE_SYSTEM_CPR"] = "1
cmake.configure(source_folder="elasticlient")"
cmake.build()

Note that you need to add all cmake.definitions calls before calling cmake.configure()

Posted by Uli Köhler in C/C++, Conan

How to fix CMake Protobuf “FindThreads only works if either C or CXX language is enabled”

Problem:

When trying to configure your CMake project using

find_package(Protobuf REQUIRED)

you see an error message like

CMake Error at /usr/share/cmake-3.16/Modules/FindThreads.cmake:49 (message):
  FindThreads only works if either C or CXX language is enabled
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindProtobuf.cmake:420 (find_package)
  CMakeLists.txt:7 (find_package)

-- Configuring incomplete, errors occurred!

Solution:

You need to put your

project(MyProject)

line before the

find_package(Protobuf REQUIRED)

line. If you don’t have a project() line, create one.

Posted by Uli Köhler in C/C++, CMake

How to compile boost unit test program using CMake

Add this to your CMakeLists.txt:

#
# Compile test suite
#
find_package(Boost COMPONENTS system filesystem unit_test_framework REQUIRED)

add_executable(test_myprogram
    tests/MyTest.cpp
)
target_include_directories(test_myprogram PUBLIC "${CMAKE_CURRENT_LIST_DIR}/include")
target_compile_features(test_myprogram PRIVATE cxx_std_17)

target_link_libraries(test_myprogram
                      ${Boost_FILESYSTEM_LIBRARY}
                      ${Boost_SYSTEM_LIBRARY}
                      ${Boost_UNIT_TEST_FRAMEWORK_LIBRARY}
                      )
add_test(test_myprogram test_myprogram)
# make "make test" run the test program
add_custom_target(test
     DEPENDS myprogram
     COMMAND ./test_myprogram
)

In order to build the program and the test, run

make

In order to run the tests, use

make test
Posted by Uli Köhler in Boost, C/C++, CMake

How to fix CMake error Could not find a package configuration file provided by “boost_unit_test_framework”

Problem:

When compiling your CMake-based C++

CMake Error at /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake:117 (find_package):
  Could not find a package configuration file provided by
  "boost_unit_test_framework" (requested version 1.71.0) with any of the
  following names:

    boost_unit_test_frameworkConfig.cmake
    boost_unit_test_framework-config.cmake

  Add the installation prefix of "boost_unit_test_framework" to
  CMAKE_PREFIX_PATH or set "boost_unit_test_framework_DIR" to a directory
  containing one of the above files.  If "boost_unit_test_framework" provides
  a separate development package or SDK, be sure it has been installed.
Call Stack (most recent call first):
  /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake:182 (boost_find_component)
  /usr/share/cmake-3.16/Modules/FindBoost.cmake:443 (find_package)
  OCCUtils/CMakeLists.txt:58 (find_package)


-- Configuring incomplete, errors occurred!

Solution:

The error message is trying to tell you that you don’t have the boost unit test library installed (or CMake can’t find it).

On Ubuntu, for example, install it using

sudo apt -y install libboost-test-dev

On Windows you can install it using the prebuilt boost binaries for Windows.

Posted by Uli Köhler in C/C++, CMake