Manage similar resources with for each

15min
|
Terraform

Terraform's for_each meta-argument allows you to configure a set of similar resources by iterating over a data structure to configure a resource or module for each item in the data structure. You can use for_each to customize a set of similar resources that share the same lifecycle.

In this tutorial, you will provision a VPC, load balancer, and EC2 instances on AWS. Then you will refactor your configuration to provision multiple projects with the for_each argument and a data structure.

Prerequisites

You can complete this tutorial using the same workflow with either Terraform Community Edition or HCP Terraform. HCP Terraform is a platform that you can use to manage and execute your Terraform projects. It includes features like remote state and execution, structured plan output, workspace resource summaries, and more.

Select the Terraform Community Edition tab to complete this tutorial using Terraform Community Edition.

This tutorial assumes that you are familiar with the Terraform and HCP Terraform workflows. If you are new to Terraform, complete Get Started collection first. If you are new to HCP Terraform, complete the HCP Terraform Get Started tutorials first.

For this tutorial, you will need:

Terraform v1.2+ installed locally.
an HCP Terraform account and organization.
HCP Terraform locally authenticated.
the AWS CLI.
an HCP Terraform variable set configured with your AWS credentials.

Apply initial configuration

Clone the example GitHub repository.

$ git clone https://github.com/hashicorp-education/learn-terraform-for-each

Change into the new directory.

$ cd learn-terraform-for-each

The configuration in main.tf provisions a VPC with public and private subnets, a load balancer, and EC2 instances in each private subnet. The variables located in variables.tf allow you to configure the VPC. For instance, the private_subnets_per_vpc variable controls the number of private subnets the configuration will create.

Set the TF_CLOUD_ORGANIZATION environment variable to your HCP Terraform organization name. This will configure your HCP Terraform integration.

$ export TF_CLOUD_ORGANIZATION=

Initialize your configuration. Terraform will automatically create the learn-terraform-for-each workspace in your HCP Terraform organization.

$ terraform initInitializing modules...Downloading registry.terraform.io/terraform-aws-modules/security-group/aws 4.9.0 for app_security_group...- app_security_group in .terraform/modules/app_security_group/modules/web- app_security_group.sg in .terraform/modules/app_security_groupDownloading registry.terraform.io/terraform-aws-modules/elb/aws 3.0.1 for elb_http...- elb_http in .terraform/modules/elb_http- elb_http.elb in .terraform/modules/elb_http/modules/elb- elb_http.elb_attachment in .terraform/modules/elb_http/modules/elb_attachmentDownloading registry.terraform.io/terraform-aws-modules/security-group/aws 4.9.0 for lb_security_group...- lb_security_group in .terraform/modules/lb_security_group/modules/web- lb_security_group.sg in .terraform/modules/lb_security_groupDownloading registry.terraform.io/terraform-aws-modules/vpc/aws 3.14.2 for vpc...- vpc in .terraform/modules/vpc Initializing HCP Terraform... Initializing provider plugins...- Reusing previous version of hashicorp/aws from the dependency lock file- Reusing previous version of hashicorp/random from the dependency lock file- Installing hashicorp/random v3.3.2...- Installed hashicorp/random v3.3.2 (signed by HashiCorp)- Installing hashicorp/aws v4.22.0...- Installed hashicorp/aws v4.22.0 (signed by HashiCorp) HCP Terraform has been successfully initialized! You may now begin working with HCP Terraform. Try running "terraform plan" tosee any changes that are required for your infrastructure. If you ever set or change modules or Terraform Settings, run "terraform init"again to reinitialize your working directory.

Note

This tutorial assumes that you are using a tutorial-specific HCP Terraform organization with a global variable set of your AWS credentials. Review the Create a Credential Variable Set for detailed guidance. If you are using a scoped variable set, assign it to your new workspace now.

Open your terraform.tf file and comment out the cloud block that configures the HCP Terraform integration.

terraform {  /*  cloud {    workspaces {      name = "learn-terraform-for-each"    }  }  */   required_providers {    aws = {      source  = "hashicorp/aws"      version = "~> 4.22.0"    }  }  required_version = ">= 1.2"}

Initialize this configuration.

$ terraform initInitializing modules...Downloading registry.terraform.io/terraform-aws-modules/security-group/aws 4.9.0 for app_security_group...- app_security_group in .terraform/modules/app_security_group/modules/web- app_security_group.sg in .terraform/modules/app_security_groupDownloading registry.terraform.io/terraform-aws-modules/elb/aws 3.0.1 for elb_http...- elb_http in .terraform/modules/elb_http- elb_http.elb in .terraform/modules/elb_http/modules/elb- elb_http.elb_attachment in .terraform/modules/elb_http/modules/elb_attachmentDownloading registry.terraform.io/terraform-aws-modules/security-group/aws 4.9.0 for lb_security_group...- lb_security_group in .terraform/modules/lb_security_group/modules/web- lb_security_group.sg in .terraform/modules/lb_security_groupDownloading registry.terraform.io/terraform-aws-modules/vpc/aws 3.14.2 for vpc...- vpc in .terraform/modules/vpc Initializing the backend... Initializing provider plugins...- Reusing previous version of hashicorp/aws from the dependency lock file- Reusing previous version of hashicorp/random from the dependency lock file- Installing hashicorp/random v3.3.2...- Installed hashicorp/random v3.3.2 (signed by HashiCorp)- Installing hashicorp/aws v4.22.0...- Installed hashicorp/aws v4.22.0 (signed by HashiCorp) Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to seeany changes that are required for your infrastructure. All Terraform commandsshould now work. If you ever set or change modules or backend configuration for Terraform,rerun this command to reinitialize your working directory. If you forget, othercommands will detect it and remind you to do so if necessary.

Once your directory has been initialized, apply the configuration, and remember to confirm with a yes.

$ terraform applyRunning apply in HCP Terraform. Output will stream here. Pressing Ctrl-Cwill cancel the remote apply if it's still pending. If the apply started itwill stop streaming the logs, but will not stop the apply running remotely. Preparing the remote apply... To view this run in a browser, visit:https://app.terraform.io/app/hashicorp-learn/learn-terraform-for-each/runs/run-b4mZpB5MQwv36ic3 Waiting for the plan to start... Terraform v1.2.4on linux_amd64Initializing plugins and modules...data.aws_availability_zones.available: Reading...data.aws_ami.amazon_linux: Reading...data.aws_availability_zones.available: Read complete after 0s [id=us-east-2]data.aws_ami.amazon_linux: Read complete after 2s [id=ami-07251f912d2a831a3] Terraform used the selected providers to generate the following executionplan. Resource actions are indicated with the following symbols:  + create Terraform will perform the following actions:   # aws_instance.app[0] will be created  + resource "aws_instance" "app" {##...Plan: 40 to add, 0 to change, 0 to destroy. Changes to Outputs:  + instance_ids    = [      + (known after apply),      + (known after apply),    ]  + public_dns_name = (known after apply)  + vpc_arn         = (known after apply) Do you want to perform these actions in workspace "learn-terraform-for-each"?  Terraform will perform the actions described above.  Only 'yes' will be accepted to approve.   Enter a value: yes random_string.lb_id: Creating...random_string.lb_id: Creation complete after 0s [id=q8SZ]module.vpc.aws_eip.nat[1]: Creating...module.vpc.aws_vpc.this[0]: Creating...module.vpc.aws_eip.nat[0]: Creating...module.vpc.aws_eip.nat[1]: Creation complete after 0s [id=eipalloc-09b056b3ef8fbe013]##...module.vpc.aws_route.private_nat_gateway[0]: Creation complete after 0s [id=r-rtb-09e62c05f9e24a2601080289494]module.vpc.aws_route.private_nat_gateway[1]: Creation complete after 0s [id=r-rtb-0eac4a0b04e4011761080289494] Apply complete! Resources: 40 added, 0 changed, 0 destroyed. Outputs: instance_ids = [  "i-0773be626797b77e9",  "i-044445cd1cd63b26c",]public_dns_name = "lb-q8SZ-client-webapp-dev-1260581575.us-east-2.elb.amazonaws.com"vpc_arn = "arn:aws:ec2:us-east-2:561656980159:vpc/vpc-0bb198242c26b103e"

Refactor the VPC and related configuration so that Terraform can deploy multiple projects at the same time, each with their own VPC and related resources.

Note

Use separate Terraform projects or workspaces instead of for_each to manage resource lifecycles independently. For example, if production and development environments share the same Terraform project running terraform destroy will destroy both.

Define a map to configure each project

Define a map for project configuration in variables.tf that for_each will iterate over to configure each resource.

variable "project" {  description = "Map of project names to configuration."  type        = map(any)   default = {    client-webapp = {      public_subnets_per_vpc  = 2,      private_subnets_per_vpc = 2,      instances_per_subnet    = 2,      instance_type           = "t2.micro",      environment             = "dev"    },    internal-webapp = {      public_subnets_per_vpc  = 1,      private_subnets_per_vpc = 1,      instances_per_subnet    = 2,      instance_type           = "t2.nano",      environment             = "test"    }  }}

Note

The for_each argument also supports lists and sets.

The project variable replaces several of the variables defined in your configuration. Remove these variable definitions from variables.tf.

-variable "project_name" {-  description = "Name of the project. Used in resource names and tags."-  type        = string-  default     = "client-webapp"-}--variable "environment" {-  description = "Value of the 'Environment' tag."-  type        = string-  default     = "dev"-}--variable "public_subnets_per_vpc" {-  description = "Number of public subnets. Maximum of 16."-  type        = number-  default     = 2-}--variable "private_subnets_per_vpc" {-  description = "Number of private subnets. Maximum of 16."-  type        = number-  default     = 2-}--variable "instance_type" {-  description = "Type of EC2 instance to use."-  type        = string-  default     = "t2.micro"-}

Add `for_each` to the VPC

Now use for_each to iterate over the project map in the VPC module block of main.tf, which will create one VPC for each key/value pair in the map.

module "vpc" {  source  = "terraform-aws-modules/vpc/aws"  version = "3.14.2"   for_each = var.project   cidr = var.vpc_cidr_block##...}

This Terraform configuration defines multiple VPCs, assigning each key/value pair in the var.project map to each.key and each.value respectively. When you use for_each with a list or set, each.key is the index of the item in the collection, and each.value is the value of the item.

In this example, the project map includes values for the number of private and public subnets in each VPC. Update the subnet configuration in the vpc module block in main.tf to use each.value to refer to these values.

module "vpc" {  source  = "terraform-aws-modules/vpc/aws"  version = "3.14.2"   for_each = var.project   cidr = var.vpc_cidr_block   azs             = data.aws_availability_zones.available.names  private_subnets = slice(var.private_subnet_cidr_blocks, 0, each.value.private_subnets_per_vpc)  public_subnets  = slice(var.public_subnet_cidr_blocks, 0, each.value.public_subnets_per_vpc)##...

Update the app_security_group module to iterate over the project variable to get the security group name, VPC ID, and CIDR blocks for each project.

module "app_security_group" {  source  = "terraform-aws-modules/security-group/aws//modules/web"  version = "4.9.0"   for_each = var.project   name        = "web-server-sg-${each.key}-${each.value.environment}"  description = "Security group for web-servers with HTTP ports open within VPC"  vpc_id      = module.vpc[each.key].vpc_id   ingress_cidr_blocks = module.vpc[each.key].public_subnets_cidr_blocks}

You can differentiate between instances of resources and modules configured with for_each by using the keys of the map you use. In this example, using module.vpc[each.key].vpc_id to define the VPC means that the security group for a given project will be assigned to the corresponding VPC.

Update the load balancer and its security group

Update the configuration for the load balancer security groups to iterate over the project variable to get their names and VPC IDs.

module "lb_security_group" {  source  = "terraform-aws-modules/security-group/aws//modules/web"  version = "4.9.0"   for_each = var.project   name = "load-balancer-sg-${each.key}-${each.value.environment}"   description = "Security group for load balancer with HTTP ports open within VPC"  vpc_id      = module.vpc[each.key].vpc_id   ingress_cidr_blocks = ["0.0.0.0/0"]}

Update the elb_http block so that each VPC's load balancer name will also include the name of the project, the environment, and will use the corresponding security groups and subnets.

module "elb_http" {  source  = "terraform-aws-modules/elb/aws"  version = "3.0.1"   for_each = var.project   # Comply with ELB name restrictions  # https://docs.aws.amazon.com/elasticloadbalancing/2012-06-01/APIReference/API_CreateLoadBalancer.html  name     = trimsuffix(substr(replace(join("-", ["lb", random_string.lb_id.result, each.key, each.value.environment]), "/[^a-zA-Z0-9-]/", ""), 0, 32), "-")  internal = false   security_groups = [module.lb_security_group[each.key].security_group_id]  subnets         = module.vpc[each.key].public_subnets##...

Move EC2 instance to a module

You will also need to update the instance resource block to assign EC2 instances to each VPC. However, the block already uses count. You cannot use both count and for_each in the same block.

To solve this, you will move the aws_instance resource into a module, including the count argument, and then use for_each when referring to the module in your main.tf file. The example repository includes a module with this configuration in the modules/aws-instance directory. For a detailed example on how to move a configuration to a local module, try the Create a Terraform Module tutorial.

Remove the resource "aws_instance" "app" and data "aws_ami" "amazon_linux" blocks from your root module's main.tf file, and replace them with a reference to the aws-instance module.

module "ec2_instances" {  source     = "./modules/aws-instance"  depends_on = [module.vpc]   for_each = var.project   instance_count     = each.value.instances_per_subnet * length(module.vpc[each.key].private_subnets)  instance_type      = each.value.instance_type  subnet_ids         = module.vpc[each.key].private_subnets[*]  security_group_ids = [module.app_security_group[each.key].security_group_id]   project_name = each.key  environment  = each.value.environment}

Note

You cannot include a provider block in modules that use count or for_each. They must inherit provider configuration from the root module. Resources created by the module will all use the same provider configuration.

Next, replace the references to the EC2 instances in the module "elb_http" block with references to the new module.

module "elb_http" {  source  = "terraform-aws-modules/elb/aws"  version = "3.0.1" ##...   number_of_instances = length(module.ec2_instances[each.key].instance_ids)  instances           = module.ec2_instances[each.key].instance_ids ##...

Finally, replace the entire contents of outputs.tf in your root module with the following.

output "public_dns_names" {  description = "Public DNS names of the load balancers for each project."  value       = { for p in sort(keys(var.project)) : p => module.elb_http[p].elb_dns_name }} output "vpc_arns" {  description = "ARNs of the vpcs for each project."  value       = { for p in sort(keys(var.project)) : p => module.vpc[p].vpc_arn }} output "instance_ids" {  description = "IDs of EC2 instances."  value       = { for p in sort(keys(var.project)) : p => module.ec2_instances[p].instance_ids }}

The for expressions used here will map the project names to the corresponding values in the Terraform output.

Note

for and for_each are different features. for_each provisions similar resources in module and resource blocks. for creates a list or map by iterating over a collection, such as another list or map. You can read more about for expressions in the Terraform documentation.

Apply scalable configuration

Initialize the new module.

$ terraform initInitializing modules...- ec2_instances in modules/aws-instance Initializing HCP Terraform... Initializing provider plugins...- Reusing previous version of hashicorp/aws from the dependency lock file- Reusing previous version of hashicorp/random from the dependency lock file- Using previously-installed hashicorp/aws v4.22.0- Using previously-installed hashicorp/random v3.3.2 HCP Terraform has been successfully initialized! You may now begin working with HCP Terraform. Try running "terraform plan" tosee any changes that are required for your infrastructure. If you ever set or change modules or Terraform Settings, run "terraform init"again to reinitialize your working directory.

Now apply the changes. Remember to respond to the confirmation prompt with yes.

$ terraform applyRunning apply in HCP Terraform. Output will stream here. Pressing Ctrl-Cwill cancel the remote apply if it's still pending. If the apply started itwill stop streaming the logs, but will not stop the apply running remotely. Preparing the remote apply... To view this run in a browser, visit:https://app.terraform.io/app/hashicorp-learn/learn-terraform-for-each/runs/run-NjEXS79EVhrVcZv1 Waiting for the plan to start... Terraform v1.2.4on linux_amd64Initializing plugins and modules...random_string.lb_id: Refreshing state... [id=q8SZ]data.aws_availability_zones.available: Reading...module.ec2_instances["internal-webapp"].data.aws_ami.amazon_linux: Reading...module.ec2_instances["client-webapp"].data.aws_ami.amazon_linux: Reading...##...Plan: 74 to add, 0 to change, 39 to destroy. Changes to Outputs:  ~ instance_ids     = [##...Do you want to perform these actions in workspace "learn-terraform-for-each"?  Terraform will perform the actions described above.  Only 'yes' will be accepted to approve.   Enter a value: yes module.lb_security_group.module.sg.aws_security_group_rule.ingress_rules[3]: Destroying... [id=sgrule-3726342134]module.app_security_group.module.sg.aws_security_group_rule.ingress_rules[2]: Destroying... [id=sgrule-3370548663]##...module.elb_http["client-webapp"].module.elb_attachment.aws_elb_attachment.this[2]: Creation complete after 0s [id=lb-q8SZ-client-webapp-dev-2022071816143201730000000a] Apply complete! Resources: 74 added, 0 changed, 39 destroyed. Outputs: instance_ids = {  "client-webapp" = [    "i-0e915495aead026a5",    "i-0aaaf5526ca78691c",    "i-0aab6ac6b7b608050",    "i-09540fa5f7a63b396",  ]  "internal-webapp" = [    "i-04fc247a327dd3a52",    "i-00aae42ee2d6f7a62",  ]}public_dns_names = {  "client-webapp" = "lb-q8SZ-client-webapp-dev-479790713.us-east-2.elb.amazonaws.com"  "internal-webapp" = "lb-q8SZ-internal-webapp-test-1463515911.us-east-2.elb.amazonaws.com"}vpc_arns = {  "client-webapp" = "arn:aws:ec2:us-east-2:561656980159:vpc/vpc-0e2c9aed33ed3eafa"  "internal-webapp" = "arn:aws:ec2:us-east-2:561656980159:vpc/vpc-0eb051e4b55cd6750"}

This configuration creates separate VPCs for each project defined in variables.tf. count and for_each allow you to create more flexible configurations, and reduce duplicate resource and module blocks.

Clean up resources

After verifying that the projects deployed successfully, run terraform destroy to destroy them. Remember to respond to the confirmation prompt with yes.

$ terraform destroyRunning apply in HCP Terraform. Output will stream here. Pressing Ctrl-Cwill cancel the remote apply if it's still pending. If the apply started itwill stop streaming the logs, but will not stop the apply running remotely. Preparing the remote apply... To view this run in a browser, visit:https://app.terraform.io/app/hashicorp-learn/learn-terraform-for-each/runs/run-sNtcVL3m2g61bopD Waiting for the plan to start... Terraform v1.2.4on linux_amd64Initializing plugins and modules...random_string.lb_id: Refreshing state... [id=q8SZ]data.aws_availability_zones.available: Reading...module.vpc["internal-webapp"].aws_vpc.this[0]: Refreshing state... [id=vpc-0eb051e4b55cd6750]##... Do you really want to destroy all resources in workspace "learn-terraform-for-each"?  Terraform will destroy all your managed infrastructure, as shown above.  There is no undo. Only 'yes' will be accepted to confirm.   Enter a value: yes module.elb_http["internal-webapp"].module.elb_attachment.aws_elb_attachment.this[0]: Destroying... [id=lb-q8SZ-internal-webapp-test-20220718161432013800000009]module.elb_http["client-webapp"].module.elb_attachment.aws_elb_attachment.this[3]: Destroying... [id=lb-q8SZ-client-webapp-dev-20220718161432004000000008]##...module.vpc["internal-webapp"].aws_eip.nat[0]: Destruction complete after 1smodule.vpc["internal-webapp"].aws_vpc.this[0]: Destruction complete after 0s Apply complete! Resources: 0 added, 0 changed, 75 destroyed.

If you used HCP Terraform for this tutorial, after destroying your resources, delete the learn-terraform-for-each workspace from your HCP Terraform organization.

Next steps

Now that you have used for_each in your configuration, explore the following resources.

Read the Terraform documentation for the for_each meta-argument.
Learn how to use the count meta-argument.
Learn how to create and use Terraform modules.

Count

CLI log in

This tutorial also appears in:

19 tutorials

Write Terraform Configuration
Learn Terraform configuration language by example. Write configurations to manage multiple pieces of infrastructure and iterate over structured data. Deploy and manage related infrastructure by referring to resources in other configurations.
- Terraform

Prerequisites

Apply initial configuration

Define a map to configure each project

Add for_each to the VPC

Update the load balancer and its security group

Move EC2 instance to a module

Apply scalable configuration

Clean up resources

Next steps

This tutorial also appears in:

Add `for_each` to the VPC