时间戳

首页 > 解决方案 > EC2 Java StartInstancesRequest 从“待处理”到“停止”再到“停止”

java - EC2 Java StartInstancesRequest 从“待处理”到“停止”再到“停止”

问题描述

我有以下情况：

我有一个 Lambda 函数，它记录 VPC 中的所有 EC2 状态更改，如下所示：

'use strict';
exports.handler = (event, context, callback) => {
  console.log('LogEC2InstanceStateChange');
  console.log('Received event:', JSON.stringify(event, null, 2));
  callback(null, 'Finished');
}

还有另一个 Lambda 函数，它尝试根据计划启动 EC2 实例，用 Java 编写，代码很多，但其核心是这样的：

public void handleRequest(Object input, Context context) {
  final List<String> instancesToStart = getInstancesToStart(); //implementation not shown
  try {
    StartInstancesRequest startRequest = new StartInstancesRequest().withInstanceIds((String[]) instancesToStart.toArray());
    context.logger.log("StartInstancesRequest: " + startRequest.toString());
    StartInstancesResult res = ec2.startInstances(startRequest);
    context.logger.log("StartInstancesResult: " + res.toString());
  }
  catch(Exception e) {
    logException(e); //calls context.logger.log on the stack trace string
  }
}

该instancesToStart数组填充有实例 ID，例如i-0abcdef1234567890.

我使用 CloudFormation 创建 Lambda 函数和所有必需的 IAM 角色等。下面是描述执行该工作的基于 Java 的 Lambda 函数的角色/权限的部分：

Resources:
  EC2SchedulerRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: /
  EC2SchedulerPolicy:
    DependsOn:
      - EC2SchedulerRole
    Type: 'AWS::IAM::Policy'
    Properties:
      PolicyName: ec2-scheduler-role
      Roles:
        - !Ref EC2SchedulerRole
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - 'logs:*'
            Resource:
              - 'arn:aws:logs:*:*:*'
          - Effect: Allow
            Action:
              - 'ec2:DescribeInstanceAttribute'
              - 'ec2:DescribeInstanceStatus'
              - 'ec2:DescribeInstances'
              - 'ec2:StartInstances'
              - 'ec2:StopInstances'
              - 'ec2:DeleteTags'
            Resource:
              - '*'

最终发生的是，根据第一个函数（记录实例状态转换的脚本）的 CloudWatch 日志，我们得到：

Received event:
{
    "version": "0",
    "id": "<guid>",
    "detail-type": "EC2 Instance State-change Notification",
    "source": "aws.ec2",
    "account": "12345678",
    "time": "2019-06-20T19:01:35Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:ec2:us-east-1:12345678:instance/i-0abcdef12345678"
    ],
    "detail": {
        "instance-id": "i-0abcdef12345678",
        "state": "pending"
    }
}

Received event:
{
    "version": "0",
    "id": "<guid>",
    "detail-type": "EC2 Instance State-change Notification",
    "source": "aws.ec2",
    "account": "12345678",
    "time": "2019-06-20T19:01:37Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:ec2:us-east-1:12345678:instance/i-0abcdef12345678"
    ],
    "detail": {
        "instance-id": "i-0abcdef12345678",
        "state": "stopping"
    }
}

Received event:
{
    "version": "0",
    "id": "<guid>",
    "detail-type": "EC2 Instance State-change Notification",
    "source": "aws.ec2",
    "account": "12345678",
    "time": "2019-06-20T19:01:37Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:ec2:us-east-1:12345678:instance/i-0abcdef12345678"
    ],
    "detail": {
        "instance-id": "i-0abcdef12345678",
        "state": "stopped"
    }
}

根据“worker”函数（实际尝试启动实例的函数）的 CloudWatch 日志，我们得到：

StartInstancesRequest: {InstanceIds: [i-0abcdef12345678],}
StartInstancesResult: {StartingInstances: [{CurrentState: {Code: 0,Name: pending},InstanceId: i-0abcdef12345678,PreviousState: {Code: 80,Name: stopped}}]}

因此，从完成工作的基于 Java 的 Lambda 的角度来看，它正在做它需要做的所有事情，发出命令使 EC2 实例启动；但是当 EC2 实例尝试实际启动时，它会从“挂起”变为“停止”再到“停止”。如果没有得到许可，它甚至不会走那么远，对吧？

如果这是实例本身（例如硬件）的问题，我希望使用 AWS 控制台手动启动它会失败。但它不会失败。手动启动成功！

那么发生了什么？我该如何进一步诊断？是权限还是实例搞砸了？

我 99% 确定这不是由于 AZ 中的可用容量不足，因为每当我尝试手动启动实例时，它总是可以工作。这不是一个短暂的问题，也不是最近才发生的事情。这种情况已经持续了几个月，其中手动启动 100% 的时间有效，而基于脚本的启动只有 0% 的时间。

标签： javaamazon-web-servicesaws-sdk

解决方案

启动 EBS 可能是问题所在。正如您所提到的，EC2 有 3 个使用 KMS 加密的 EBS 卷。您必须提供 KMS 权限(kms:CreateGrant) 才能启动您的实例

{
        "Sid": "GrantAccess",
        "Effect": "Allow",
        "Action": "kms:CreateGrant",
        "Resource": "arn:aws:kms:::key/1234"
}

推荐阅读