How to fix Quay PermissionError: [Errno 13] Permission denied: '/datastorage/registry'

Problem

While uploading to a Quay registry, the upload process is continously Retrying and the quay logs shows the following error:

Traceback (most recent call last):
  File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 65, in handle
    util.reraise(*sys.exc_info())
  File "/app/lib/python3.9/site-packages/gunicorn/util.py", line 626, in reraise
    raise value
  File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 55, in handle
    self.handle_request(listener_name, req, client, addr)
  File "/app/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 127, in handle_request
    super().handle_request(listener_name, req, sock, addr)
  File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 130, in handle_request
    util.reraise(*sys.exc_info())
  File "/app/lib/python3.9/site-packages/gunicorn/util.py", line 626, in reraise
    raise value
  File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/app/lib/python3.9/site-packages/flask/app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "/app/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 169, in __call__
    return self.app(environ, start_response)
  File "/app/lib/python3.9/site-packages/flask/app.py", line 2449, in wsgi_app
    response = self.handle_exception(e)
  File "/app/lib/python3.9/site-packages/flask/app.py", line 1866, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/app/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/app/lib/python3.9/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/app/lib/python3.9/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/app/lib/python3.9/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/app/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/app/lib/python3.9/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/app/lib/python3.9/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/quay-registry/endpoints/decorators.py", line 227, in wrapper
    return func(*args, **kwargs)
  File "/quay-registry/endpoints/decorators.py", line 90, in wrapper
    return func(*args, **kwargs)
  File "/quay-registry/auth/registry_jwt_auth.py", line 175, in wrapper
    return func(*args, **kwargs)
  File "/quay-registry/endpoints/v2/__init__.py", line 169, in wrapped
    return func(namespace_name, repo_name, *args, **kwargs)
  File "/quay-registry/endpoints/decorators.py", line 163, in wrapper
    return func(*args, **kwargs)
  File "/quay-registry/endpoints/decorators.py", line 188, in wrapper
    return func(*args, **kwargs)
  File "/quay-registry/endpoints/v2/blob.py", line 268, in start_blob_upload
    blob_uploader = create_blob_upload(repository_ref, storage, _upload_settings())
  File "/quay-registry/data/registry_model/blobuploader.py", line 74, in create_blob_upload
    new_upload_uuid, upload_metadata = storage.initiate_chunked_upload(location_name)
  File "/quay-registry/storage/distributedstorage.py", line 26, in wrapper
    return storage_func(*args, **kwargs)
  File "/quay-registry/storage/local.py", line 90, in initiate_chunked_upload
    with open(self._init_path(self._rel_upload_path(new_uuid), create=True), "wb"):
  File "/quay-registry/storage/local.py", line 25, in _init_path
    os.makedirs(dirname)
  File "/usr/lib64/python3.9/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib64/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/datastorage/registry'

Solution

In your docker-compose.yml for quay, find which directory is mapped onto /storage

services:
  quay:
    image: quay.io/projectquay/quay:latest
    # [...]
    volumes:
      - ./quay_storage:/datastorage # <--

For this example, the directory ./quay_storage is mapped to /datastorage in the container.

In this directory, create the registry sub-directory:

mkdir -p ./quay_storage/registry
chmod -R 777 ./quay_storage

I haven’t quite figured out what the correct permissions are, so be aware of the security implications of chmod -R 777 which makes the directory and all its contents world-readable and world-writable.