gitlab - Gitlab CE 容器注册表超时 - Rack::Timeout::RequestTimeoutException

Question

我想请你帮忙，我有gitlab-ce实例在 AWS EC2 上运行，作为独立的 GitLab，并且容器（docker）注册表不工作。谢谢你。

问题：在 Gitlab CE 版本（自托管）中，容器注册表不工作。症状：

1. 在任何项目上通过 UI 访问 GitLab 注册表都会导致 500 HTTP 错误“哎呀，我们这边出了点问题。”

硬数据：

1. gitlab-ce已经是最新版本(13.3.5-ce.0) - 通过 GitLab 综合安装从官方上游安装。
2. gitlab.rb相关部分 - 如您所见，所有内容均已注释掉
3. 访问注册表的用户是实例的管理员

################################################################################
## Container Registry settings
##! Docs: https://docs.gitlab.com/ee/administration/container_registry.html
################################################################################

registry_external_url 'https://gitlab.mydomain.com:5005'

### Settings used by GitLab application
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "gitlab.mydomain.com"
gitlab_rails['registry_port'] = "5005"
# gitlab_rails['registry_path'] = "/mnt/gitlab/gitlab-rails/shared/registry"

# Notification secret, it's used to authenticate notification requests to GitLab application
# You only need to change this when you use external Registry service, otherwise
# it will be taken directly from notification settings of your Registry
# gitlab_rails['registry_notification_secret'] = nil

分析：

1. 在日志和转储跟踪之后，此堆栈跟踪存在于日志中的超时限制上：

Rack::Timeout::RequestTimeoutException (Request ran for longer than 60000ms):
  
lib/container_registry/client.rb:45:in `repository_tags'
app/models/container_repository.rb:63:in `manifest'
lib/gitlab/metrics/instrumentation.rb:161:in `block in manifest'
lib/gitlab/metrics/method_call.rb:27:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `manifest'
app/models/container_repository.rb:67:in `tags'
lib/gitlab/metrics/instrumentation.rb:161:in `block in tags'
lib/gitlab/metrics/method_call.rb:27:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `tags'
app/models/container_repository.rb:87:in `has_tags?'
lib/gitlab/metrics/instrumentation.rb:161:in `block in has_tags?'
lib/gitlab/metrics/method_call.rb:27:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `has_tags?'
app/controllers/projects/registry/repositories_controller.rb:57:in `block (2 levels) in ensure_root_container_repository!'
app/controllers/projects/registry/repositories_controller.rb:56:in `tap'
app/controllers/projects/registry/repositories_controller.rb:56:in `block in ensure_root_container_repository!'
app/controllers/projects/registry/repositories_controller.rb:53:in `tap'
app/controllers/projects/registry/repositories_controller.rb:53:in `ensure_root_container_repository!'
app/controllers/application_controller.rb:491:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:482:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
app/controllers/application_controller.rb:476:in `set_locale'
lib/gitlab/error_tracking.rb:50:in `with_context'
app/controllers/application_controller.rb:541:in `sentry_context'
app/controllers/application_controller.rb:469:in `block in set_current_context'
lib/gitlab/application_context.rb:52:in `block in use'
lib/gitlab/application_context.rb:52:in `use'
lib/gitlab/application_context.rb:20:in `with_context'
app/controllers/application_controller.rb:462:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/transaction.rb:61:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/multipart.rb:140:in `call'
lib/gitlab/middleware/read_only/controller.rb:51:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

有关更多详细信息，请参阅此 json

{
    "method": "GET",
    "path": "## REDACTED ##/container_registry",
    "format": "html",
    "controller": "Projects::Registry::RepositoriesController",
    "action": "index",
    "status": 500,
    "time": "2020-09-10T10:48:28.546Z",
    "params": [{
        "key": "namespace_id",
        "value": "## REDACTED ##"
    }, {
        "key": "project_id",
        "value": "## REDACTED ##"
    }],
    "remote_ip": "## REDACTED ##",
    "user_id": 1,
    "username": "## REDACTED ##",
    "ua": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36",
    "correlation_id": "9xxF4mcYPP6",
    "meta.user": "## REDACTED ##",
    "meta.project": "## REDACTED ##",
    "meta.root_namespace": "## REDACTED ##",
    "meta.caller_id": "Projects::Registry::RepositoriesController#index",
    "redis_calls": 11,
    "redis_duration_s": 0.0033759999999999997,
    "redis_read_bytes": 1568,
    "redis_write_bytes": 1664,
    "redis_cache_calls": 10,
    "redis_cache_duration_s": 0.002717,
    "redis_cache_read_bytes": 1387,
    "redis_cache_write_bytes": 548,
    "redis_shared_state_calls": 1,
    "redis_shared_state_duration_s": 0.000659,
    "redis_shared_state_read_bytes": 181,
    "redis_shared_state_write_bytes": 1116,
    "queue_duration_s": 0.026687,
    "cpu_s": 0.09,
    "exception.class": "Rack::Timeout::RequestTimeoutException",
    "exception.message": "Request ran for longer than 60000ms",
    "exception.backtrace": 
    [
        "lib/container_registry/client.rb:45:in `repository_tags'", 
        "app/models/container_repository.rb:63:in `manifest'", 
        "lib/gitlab/metrics/instrumentation.rb:161:in `block in manifest'", 
        "lib/gitlab/metrics/method_call.rb:27:in `measure'", 
        "lib/gitlab/metrics/instrumentation.rb:161:in `manifest'", 
        "app/models/container_repository.rb:67:in `tags'", 
        "lib/gitlab/metrics/instrumentation.rb:161:in `block in tags'", 
        "lib/gitlab/metrics/method_call.rb:27:in `measure'", 
        "lib/gitlab/metrics/instrumentation.rb:161:in `tags'", 
        "app/models/container_repository.rb:87:in `has_tags?'", 
        "lib/gitlab/metrics/instrumentation.rb:161:in `block in has_tags?'", 
        "lib/gitlab/metrics/method_call.rb:27:in `measure'", 
        "lib/gitlab/metrics/instrumentation.rb:161:in `has_tags?'", 
        "app/controllers/projects/registry/repositories_controller.rb:57:in `block (2 levels) in ensure_root_container_repository!'", 
        "app/controllers/projects/registry/repositories_controller.rb:56:in `tap'", 
        "app/controllers/projects/registry/repositories_controller.rb:56:in `block in ensure_root_container_repository!'", 
        "app/controllers/projects/registry/repositories_controller.rb:53:in `tap'", 
        "app/controllers/projects/registry/repositories_controller.rb:53:in `ensure_root_container_repository!'", 
        "app/controllers/application_controller.rb:491:in `set_current_admin'", 
        "lib/gitlab/session.rb:11:in `with_session'", 
        "app/controllers/application_controller.rb:482:in `set_session_storage'", 
        "lib/gitlab/i18n.rb:73:in `with_locale'", 
        "lib/gitlab/i18n.rb:79:in `with_user_locale'", 
        "app/controllers/application_controller.rb:476:in `set_locale'", 
        "lib/gitlab/error_tracking.rb:50:in `with_context'", 
        "app/controllers/application_controller.rb:541:in `sentry_context'", 
        "app/controllers/application_controller.rb:469:in `block in set_current_context'", 
        "lib/gitlab/application_context.rb:52:in `block in use'",
        "lib/gitlab/application_context.rb:52:in `use'", 
        "lib/gitlab/application_context.rb:20:in `with_context'", 
        "app/controllers/application_controller.rb:462:in `set_current_context'", 
        "lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'", 
        "lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'", 
        "lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'", 
        "lib/gitlab/metrics/transaction.rb:61:in `run'", 
        "lib/gitlab/metrics/rack_middleware.rb:16:in `call'", 
        "lib/gitlab/request_profiler/middleware.rb:17:in `call'", 
        "lib/gitlab/middleware/go.rb:20:in `call'", 
        "lib/gitlab/etag_caching/middleware.rb:13:in `call'", 
        "lib/gitlab/middleware/multipart.rb:140:in `call'", 
        "lib/gitlab/middleware/read_only/controller.rb:51:in `call'", 
        "lib/gitlab/middleware/read_only.rb:18:in `call'", 
        "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'", 
        "lib/gitlab/middleware/basic_health_check.rb:25:in `call'", 
        "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'", 
        "lib/gitlab/middleware/request_context.rb:23:in `call'",
        "config/initializers/fix_local_cache_middleware.rb:9:in `call'",
        "lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call'",
        "lib/gitlab/middleware/release_env.rb:12:in `call'"],
    "db_duration_s": 0.00385,
    "view_duration_s": 0.0,
    "duration_s": 59.94318,
    "db_count": 10,
    "db_write_count": 0,
    "db_cached_count": 1
}

2. 注册表应该是空的，但是运行gitlab-ctl registry-garbage-collect仍然会导致错误

Running garbage-collect using configuration ["/opt/gitlab/embedded/bin/registry", "garbage-collect", "/var/opt/gitlab/registry/config.yml"], this might take a while...
INFO[0000] starting mark stage                           environment=production go.version=go1.14.7 instance.id=d714caeb-b88f-4bca-823e-9da27acb9999 service=registry
failed to garbage collect: failed to mark: : Path not found: /docker/registry/v2/repositories
Failed to run garbage-collect command, starting registry service.

3. Gitlab Registry API 响应 localhost:5000/v2/ 地址的 curl （默认一个）

# curl localhost:5000/v2/      
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

标签： gitlabdocker-registrygitlab-omnibusgitlab-ce