时间戳

首页 > 解决方案 > 请求计数的增加以“错误：套接字挂起”结束

http - 请求计数的增加以“错误：套接字挂起”结束

问题描述

当请求数量增加时，后端服务/nginx 代理开始响应“错误：套接字挂起”。设置如下。

操作系统：CentOS 6

Express JS 服务 -> nginx 作为代理 -> Gunicorn 运行的烧瓶应用程序

JS 应用程序同时向另一个服务发送多个请求，当请求计数超过 ~100 时，它开始返回错误响应。如果计数较低，则一切正常。

我遵循了 Gunicorn 文档中的 nginx 示例配置 + 增加超时限制 + 增加 nginx 打开文件限制。我也尝试过keepalive选项，但问题仍然存在。Gunicorn 没有显示任何错误。

nginx配置片段：

upstream app_server {
    server 127.0.0.1:8000 fail_timeout=0;
    keepalive 100;
}

server {
    listen 5001;
    client_max_body_size 4G;

    keepalive_timeout 300;

    root /path/to/app/current/public; # static files

    location / {
        try_files $uri @proxy_to_app;
    }

    location @proxy_to_app {
        # Timeouts
        proxy_read_timeout 300;
        proxy_connect_timeout 300;
        proxy_send_timeout 300;
        send_timeout 300;

        proxy_http_version 1.1;

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://app_server;
    }
}

从代理收到的错误响应：

{ RequestError: Error: socket hang up
    at new RequestError (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/errors.js:14:15)
    at Request.plumbing.callback (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/plumbing.js:87:29)
    at Request.RP$callback [as _callback] (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/plumbing.js:46:31)
    at self.callback (/home/pm2deploy/apps/app-backend/source/node_modules/request/request.js:185:22)
    at Request.emit (events.js:160:13)
    at Request.onRequestError (/home/pm2deploy/apps/app-backend/source/node_modules/request/request.js:881:8)
    at ClientRequest.emit (events.js:160:13)
    at Socket.socketOnEnd (_http_client.js:423:9)
    at Socket.emit (events.js:165:20)
    at endReadableNT (_stream_readable.js:1101:12)
    at process._tickCallback (internal/process/next_tick.js:152:19)
  name: 'RequestError',
  message: 'Error: socket hang up',
  cause: { Error: socket hang up
    at createHangUpError (_http_client.js:330:15)
    at Socket.socketOnEnd (_http_client.js:423:23)
    at Socket.emit (events.js:165:20)
    at endReadableNT (_stream_readable.js:1101:12)
    at process._tickCallback (internal/process/next_tick.js:152:19) code: 'ECONNRESET' },
  error: { Error: socket hang up
    at createHangUpError (_http_client.js:330:15)
    at Socket.socketOnEnd (_http_client.js:423:23)
    at Socket.emit (events.js:165:20)
    at endReadableNT (_stream_readable.js:1101:12)
    at process._tickCallback (internal/process/next_tick.js:152:19) code: 'ECONNRESET' },
  options:
   { method: 'PUT',
     uri: 'http://localhost:5001/transformers/segmentAvg',
     qs:
      { stdMultiplier: 2,
        segmentLeft: 1509366682333,
        segmentRight: 1509367401685 },
     body: { index: [Array], values: [Array] },
     headers: {},
     json: true,
     callback: [Function: RP$callback],
     transform: undefined,
     simple: true,
     resolveWithFullResponse: false,
     transform2xxOnly: false },
  response: undefined }

添加：

在 OS 日志中记录了以下条目：

possible SYN flooding on port X. Sending cookies.

标签： httpnginxflaskgunicorncentos6

解决方案

内核套接字积压达到限制并丢弃了以下请求。

原因：在 Red Hat Enterprise Linux 中，由于 LISTEN 套接字缓冲区已满，内核丢弃 TCP 连接

增加内核套接字积压限制

检查当前值：

# sysctl net.core.somaxconn
net.core.somaxconn = 128

增加值：

# sysctl -w net.core.somaxconn=2048
net.core.somaxconn = 2048

通过再次查看确认更改：

# sysctl net.core.somaxconn
net.core.somaxconn = 2048

坚持改变：

echo "net.core.somaxconn = 2048" >> /etc/sysctl.conf

增加应用程序套接字侦听积压

uWSGI的配置参数

listen=1024

此解决方案取自https://access.redhat.com/solutions/30453

推荐阅读