terraform - 避免使用 Terraform 重新创建 EMR 集群

问题描述

此问题与 Terraform 0.12版本有关

你好！我目前正在使用 Terraform 在我的团队中部署 AWS EMR 集群，每次我们添加新资源时，Terraform 都会重新创建集群。我们集群的声明如下：

resource "aws_emr_cluster" "emr_spark_cluster" {
  name = "whatever_name_to_emr_spark_cluster"
  release_label = "emr-5.29.0"
  applications = ["Spark"]
  log_uri = "whatever_log_uri"

  ec2_attributes {
    subnet_id = var.vpc_public_subnets[0]
    emr_managed_master_security_group = aws_security_group.emr_sg.id
    emr_managed_slave_security_group = aws_security_group.emr_sg.id
    instance_profile = aws_iam_instance_profile.instance_profile.arn
    key_name = "whatever_name_to_ec2_attributes"
  }

  master_instance_group {
    instance_type = var.emr_master_instance_type
  }

  core_instance_group {
    instance_type = var.emr_core_instance_type
    instance_count = var.emr_core_instance_count

    ebs_config {
      size = var.emr_core_ebs_size
      type = whatever_type
      volumes_per_instance = whatever_volumes_per_instance
    }
  }

  tags = var.tags
  service_role = aws_iam_role.emr_service.arn
}

基本上我们使用master_instance_group和core_instance_group作为Terraform 0.12 文档建议的( 1 )

随着集群已经部署并尝试添加新资源，terraform apply命令说需要重新创建集群（删除所有不感兴趣的东西）：

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # module.emr.aws_emr_cluster.emr_spark_cluster must be replaced
-/+ resource "aws_emr_cluster" "emr_spark_cluster" {
        applications                      = [whatever_applications]]
      - ebs_root_volume_size              = 0 -> null

      - instance_group {
          - id             = whatever_id -> null
          - instance_count = 2 -> null
          - instance_role  = "CORE" -> null
          - instance_type  = whatever_the_instance_type -> null

          - ebs_config {
              - iops                 = 0 -> null
              - size                 = whatever_size -> null
              - type                 = whatever_type -> null
              - volumes_per_instance = whatever_volumes_per_instance -> null
            }
        }
      - instance_group {
          - id             = whatever_id -> null
          - instance_count = whatever_instance_count -> null
          - instance_role  = "MASTER" -> null
          - instance_type  = whatever_the_instance_type -> null

          - ebs_config {
              - iops                 = 0 -> null
              - size                 = whatever_size -> null
              - type                 = whatever_type -> null
              - volumes_per_instance = whatever_volumes_per_instance -> null
            }
        }
      + instance_group {
          + autoscaling_policy = (known after apply)
          + bid_price          = (known after apply)
          + id                 = (known after apply)
          + instance_count     = (known after apply)
          + instance_role      = (known after apply)
          + instance_type      = (known after apply)
          + name               = (known after apply)

          + ebs_config {
              + iops                 = (known after apply)
              + size                 = (known after apply)
              + type                 = (known after apply)
              + volumes_per_instance = (known after apply)
            }
        }

      ~ master_instance_group {
          ~ id             = whatever_id -> (known after apply)
            instance_count = 1
            instance_type  = whatever_the_instance_type

          - ebs_config {
              - iops                 = 0 -> null
              - size                 = whatever_size -> null
              - type                 = whatever_type -> null
              - volumes_per_instance = whatever_volumes_per_instance -> null
            }
          + ebs_config {
              + iops                 = (known after apply)
              + size                 = (known after apply)
              + type                 = (known after apply)
              + volumes_per_instance = (known after apply)
            }
        }
    }

基本上，我真正感兴趣的部分是instance_group. 我想这就是为什么需要重新创建集群的原因，因为当前master_instance_group和core_instance_group属性不会“保留”特定的实例组，而只是它们的类型和大小，对吧？

出于这个原因，作为一种解决方法，我尝试使用aws_emr_instance_group资源尝试替换master_instance_group和core_instance_group属性并通过属性将它们直接链接到集群cluster_id。例如，aws_emr_instance_group的master_instances：

resource "aws_emr_instance_group" "master_instances" {
  name = "emr_cluster_master_instances"
  cluster_id = aws_emr_cluster.emr_spark_cluster.id
  instance_type = var.emr_master_instance_type
  instance_count = var.emr_master_instance_count
  ebs_config {
      iops = 0
      size = var.emr_core_ebs_size
      type = whatever_type
      volumes_per_instance = whatever_volumes_per_instance
  }
}

但在执行时terraform plan它失败并出现以下错误：

module.emr.aws_emr_cluster.emr_spark_cluster: Creating...

Error: error running EMR Job Flow: ValidationException: Instance count must be greater than 0

我不明白这个错误，因为我正在将实例组与集群链接起来。

所以我的问题是：

如何为我的集群设置特定实例以避免每次我需要添加新资源时重新创建它（在另一个管道中部署新资源被丢弃）。我是否必须使用master_instance_type，即使这些参数已被弃用core_instance_type？core_instance_count也许我错误地使用了aws_emr_instance_group资源？
有没有可能这可能是属性master_instance_group和的预期行为core_instance_group？我的意思是，即使集群被破坏，也要避免创建特定的实例以使其处于活动状态？

提前致谢

( 1 )我说“建议”是因为在 Terraform 0.12 中不推荐使用其他属性，例如core_instance_type,core_instance_count和master_instance_type。

标签： terraformamazon-emr

terraform - 避免使用 Terraform 重新创建 EMR 集群

问题描述

解决方案

推荐阅读