Integrate service mesh and API gateway

10min
|
Nomad
Consul

Consul service mesh is a feature that provides secure service-to-service communication with Transport Layer Security (TLS). In a service mesh, sidecar proxies run alongside each service instance, allowing the service to send requests to localhost instead of a defined upstream service address. All traffic sent to each application service travels through an outbound and inbound sidecar proxy. As a result, each service has encrypted communication and Consul enforces application routing rules with service intentions.

The next step of this monolith migration integrates Consul service mesh to allow secure TLS connections between each service. It also sets up an API gateway for external public access.

In this tutorial, you deploy a version of HashiCups that uses Consul service mesh while running on private client nodes. Then you deploy an API gateway that allows external public access the services running on the private client node.

Infrastructure overview

At the beginning of the tutorial you have a Nomad and Consul cluster with three server nodes, three private client nodes, and one publicly accessible client node. Each node runs a Consul agent and a Nomad agent.

Architectural diagram. Consul and Nomad cluster with one public client and three private clients.

This infrastructure matches the end state of the previous tutorial.

When you complete the steps in this tutorial, HashiCups is secured by the Consul service mesh and accessible using the API gateway.

Architectural diagram. HashiCups running in Consul service mesh and accessible using Consul API gateway.

This version of HashiCups has the following attributes:

All services now run on Nomad private client nodes
Consul service mesh integration with upstream service configurations
An API gateway running on the Nomad public client node
Nomad Workload Identity to automatically generate ACL tokens for the API gateway
Consul service intentions
Secure connections with custom TLS certificates and mTLS with Envoy proxies. Services in Consul service mesh connect to upstreams using the localhost address.

Prerequisites

This tutorial uses the infrastructure set up in a previous tutorial in this collection, Set up the cluster. Complete that tutorial to set up the infrastructure if you have not done so.

Review configuration files

In your terminal, navigate to the directory that contains the code from the learn-consul-nomad-vm code repository.

Navigate to the jobs directory.

$ cd shared/jobs

Additional files for configuring the API gateway and service intentions are located in the shared/jobs directory. These files include 04.api-gateway.config.sh, 04.api-gateway.nomad.hcl, and 04.intentions.consul.sh.

Review scripts for API gateway configuration

The Consul API gateway allows external traffic to the internal nginx service.

The repository provides a script, 04.api-gateway.config.sh, that automates the initial configuration required for Nomad and Consul.

Click here to examine the details of the configuration script.

The set up script starts by cleaning up previous Consul configurations, binding rules, auth methods, and the ingress namespace in Nomad.

echo -e "${_COL}Clean previous configurations.${_NC}" # Remove route for NGINXconsul config delete -kind http-route -name hashicups-http-route # Remove Inline certificateconsul config delete -kind inline-certificate -name api-gw-certicate # Remove API Gateway Listenerconsul config delete -kind api-gateway -name api-gateway # Remove all existing binding rules# WARNING: if you have existing binding rules you want to maintain, modify this behaviorfor i in `consul acl binding-rule list -format json | jq -r .[].ID`; do  consul acl binding-rule delete -id=$idone # Delete Nomad namespacenomad namespace delete ingress # Delete Consul auth-methodconsul acl auth-method delete -name nomad-workloads if [ "$1 " == "-clean " ]; then   echo -e "${_ERR}Only cleaning selected...Exiting.${_NC}"  exit 0  fi# ...

Then the script configures the Consul and Nomad ACL integration, which includes setting up the auth method, creating a binding rule, and creating the ingress namespace in Nomad.

# ...tee ${_JWT_FILE} > /dev/null << EOF{  # ...}EOF# This auth method creates an endpoint for generating Consul ACL tokens #  from Nomad workload identities.consul acl auth-method create \            -name 'nomad-workloads' \            -type 'jwt' \            -description 'JWT auth-method for Nomad services and workloads' \            -config "@${_JWT_FILE}"# ...# The 'ingress' namespace will be used to deploy the API Gateway and to identify#   Nomad Jobs that require a token to be generated automatically. nomad namespace apply \    -description "namespace for Consul API Gateways" \    ingress # The binding-rule identifies all Nomad Jobs in the 'ingress' namespaces and #  uses the 'builtin/api-gateway' policy to generate a Consul ACL token for them.# ... tee ${_BR_FILE} > /dev/null << EOF{  # ...}EOF curl --silent \  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \  --connect-to ${CONSUL_TLS_SERVER_NAME}:8443:${_consul_addr} \  --cacert ${CONSUL_CACERT} \  --data @${_BR_FILE} \  --request PUT \  https://${CONSUL_TLS_SERVER_NAME}:8443/v1/acl/binding-rule | jq

Finally, the script generates TLS certificates, creates a listener and route for the API gateway, and then writes them to Consul configurations.

# ...# Generate TLS certificatestee ${_ssl_conf_FILE} > /dev/null << EOF# ...EOF openssl genrsa -out ${_ssl_key_file}  4096 2>/dev/null openssl req -new \  -key ${_ssl_key_file} \  -out ${_ssl_csr_file} \  -config ${_ssl_conf_FILE} 2>/dev/null openssl x509 -req -days 3650 \  -in ${_ssl_csr_file} \  -signkey ${_ssl_key_file} \  -out ${_ssl_crt_file} 2>/dev/null export API_GW_KEY=`cat ${_ssl_key_file}`export API_GW_CERT=`cat ${_ssl_crt_file}` # Create 'api-gateway-certificate' inline-certificate"tee ${_GW_certificate_FILE} > /dev/null << EOF# ...EOF consul config write ${_GW_certificate_FILE} # Create 'api-gateway' HTTP listener on port 8443tee ${_GW_config_FILE} > /dev/null << EOF# ...EOF consul config write ${_GW_config_FILE} # Create 'hashicups-http-route' HTTP route "/" > "nginx"tee ${_GW_route_FILE} > /dev/null << EOF# ...EOF consul config write ${_GW_route_FILE}

Review the API gateway jobspec

The API gateway job is constrained to a node with the ingress meta role that also runs in the ingress namespace. It has four tasks: two are for job set up and clean up, one registers the gateway with Consul, and one starts the gateway. The set up, clean up, and service registration tasks each define a lifecycle block that enforces the order in which they run.

# ...job "api-gateway" {   namespace = var.namespace   constraint {    attribute = "${meta.nodeRole}"    operator  = "="    value     = "ingress"  }  group "api-gateway" {  # ...  task "setup" {    driver = "docker"     config {      image = var.consul_image # image containing Consul      command = "/bin/sh"      args = [        "-c",        "consul connect envoy -gateway api -register -deregister-after-critical 10s -service ${NOMAD_JOB_NAME} -admin-bind 0.0.0.0:19000 -ignore-envoy-compatibility -bootstrap > ${NOMAD_ALLOC_DIR}/envoy_bootstrap.json"      ]    }     lifecycle {      hook = "prestart"      sidecar = false    }    # ...  }  task "api-gw" {    # ...    config {      image = var.envoy_image # image containing Envoy      args = [        "--config-path",        "${NOMAD_ALLOC_DIR}/envoy_bootstrap.json",        "--log-level",        "${meta.connect.log_level}",        "--concurrency",        "${meta.connect.proxy_concurrency}",        "--disable-hot-restart"      ]    }  }  task "service_change" {    # ...    lifecycle {      hook = "poststart"    }    # ...    config {      command = "consul"      args    = ["services", "register", "${NOMAD_TASK_DIR}/svc-api-gateway.hcl"]    }     template {      data = <<EOF        service {          id      = "api-gateway"          name    = "api-gateway"          kind    = "api-gateway"          port    = 8443          address = ""          meta = {            public_address = "https://{{ env "attr.unique.platform.aws.public-ipv4" }}:8443"          }        }      EOF       destination = "${NOMAD_TASK_DIR}/svc-api-gateway.hcl"    }  }  task "cleanup" {      # ...      lifecycle {        hook = "poststop"      }      # ...      config {        command = "consul"        args    = ["services", "deregister", "-id=api-gateway"]      }    }  }

Review the Consul intentions configuration

In a secure configuration, Consul uses intentions to specify allowed communications across services.

The repository provides a script, 04.intentions.consul.sh, that automates the creation of intentions.

Click here to examine the details of the configuration script.

The Consul intentions script cleans up any previous configurations and then creates intentions for the HashiCups services. Intentions are how Consul enforces routes of communication between services.

## ... ##-------------------------------------------------------------------------------## Clean previous configurations##------------------------------------------------------------------------------- echo -e "${_COL}Clean previous configurations.${_NC}" consul config delete -kind service-intentions -name databaseconsul config delete -kind service-intentions -name product-apiconsul config delete -kind service-intentions -name payments-apiconsul config delete -kind service-intentions -name public-apiconsul config delete -kind service-intentions -name frontendconsul config delete -kind service-intentions -name nginx ## ... ### ----------------------------------------------------------------------------### Configure Consul Intentions### ---------------------------------------------------------------------------- ## References:## - https://developer.hashicorp.com/consul/docs/connect/config-entries/service-intentions echo -e "${_COL}Create Consul intentions for Hashicups and API Gateway${_NC}" tee ${_int_DB_FILE} > /dev/null << EOFKind = "service-intentions"Name = "database"Sources = [  {    Name   = "product-api"    Action = "allow"  }]EOF tee ${_int_PROD_API_FILE} > /dev/null << EOFKind = "service-intentions"Name = "product-api"Sources = [  {    Name   = "public-api"    Action = "allow"  }]EOF tee ${_int_PAY_API_FILE} > /dev/null << EOFKind = "service-intentions"Name = "payments-api"Sources = [  {    Name   = "public-api"    Action = "allow"  }]EOF  tee ${_int_PUB_API_FILE} > /dev/null << EOFKind = "service-intentions"Name = "public-api"Sources = [  {    Name   = "nginx"    Action = "allow"  }]EOF tee ${_int_FE_FILE} > /dev/null << EOFKind = "service-intentions"Name = "frontend"Sources = [  {    Name   = "nginx"    Action = "allow"  }]EOF tee ${_int_NGINX_FILE} > /dev/null << EOFKind = "service-intentions"Name = "nginx"Sources = [  {    Name   = "api-gateway"    Action = "allow"  }]EOF consul config write ${_int_DB_FILE}consul config write ${_int_PROD_API_FILE}consul config write ${_int_PAY_API_FILE}consul config write ${_int_PUB_API_FILE}consul config write ${_int_FE_FILE}consul config write ${_int_NGINX_FILE}

Review the HashiCups jobspec

Open the 04.hashicups.nomad.hcl jobspec file and review the contents.

This version uses the same constraint as previous versions to run all of the HashiCups services on private client nodes. It integrates Consul service mesh with upstream configurations and configures services to use mTLS through the Envoy sidecar proxy. These configuration attributes are located in the connect block.

job "hashicups" {  # ...  group "db" {    # ...    service {      # ...      connect {        sidecar_service {}      }      # ...    }    # ...  }  # ...  group "public-api" {    # ...    service {      # ...      connect {        sidecar_service {          proxy {            upstreams {              destination_name = "product-api"              local_bind_port = 9090            }            upstreams {              destination_name = "payments-api"              local_bind_port = 8080            }          }        }      }      # ...    }    # ...  }  # ...}

Deploy Consul API gateway

Set up and deploy the Consul API gateway before you submit the updated jobspec to Nomad. Set up and deploy the Consul API gateway before you submit the updated jobspec to Nomad.

Run the API setup script and jobspec

Set up the API gateway configurations in Consul.

$ ./04.api-gateway.config.shConfigure environment.Clean previous configurations.Config entry deleted: http-route/hashicups-http-routeConfig entry deleted: inline-certificate/api-gw-certicateConfig entry deleted: api-gateway/api-gatewayError deleting namespace: Unexpected response code: 500 (rpc error: namespace not found)Error deleting auth method "nomad-workloads": Unexpected response code: 404 (Cannot find auth method to delete) ## ... Create Nomad namespace 'ingress'Successfully applied namespace "ingress"!Create Consul binding-rule 'Nomad API gateway' ## ... Generate TLS certificate for 'hashicups.hashicorp.com'Create 'api-gateway-certificate' inline-certificateConfig entry written: inline-certificate/api-gw-certificateCreate 'api-gateway' HTTP listener on port 8443Config entry written: api-gateway/api-gatewayCreate 'hashicups-http-route' HTTP route '/' > 'nginx'Config entry written: http-route/hashicups-http-route

Submit the API gateway job to Nomad.

$ nomad job run 04.api-gateway.nomad.hcl==> 2024-11-13T15:21:40+01:00: Monitoring evaluation "f25cae62"    2024-11-13T15:21:40+01:00: Evaluation triggered by job "api-gateway"    2024-11-13T15:21:40+01:00: Allocation "c1f36918" created: node "30b5f033", group "api-gateway"    2024-11-13T15:21:42+01:00: Evaluation within deployment: "796699a7"    2024-11-13T15:21:42+01:00: Evaluation status changed: "pending" -> "complete"==> 2024-11-13T15:21:42+01:00: Evaluation "f25cae62" finished with status "complete"==> 2024-11-13T15:21:42+01:00: Monitoring deployment "796699a7"  ✓ Deployment "796699a7" successful     2024-11-13T15:22:03+01:00    ID          = 796699a7    Job ID      = api-gateway    Job Version = 0    Status      = successful    Description = Deployment completed successfully     Deployed    Task Group   Desired  Placed  Healthy  Unhealthy  Progress Deadline    api-gateway  1        1       1        0          2024-11-13T14:32:00Z

Run the Consul intentions setup script

Set up the service intentions in Consul.

$ ./04.intentions.consul.shConfigure environment.Clean previous configurations.Config entry deleted: service-intentions/databaseConfig entry deleted: service-intentions/product-apiConfig entry deleted: service-intentions/payments-apiConfig entry deleted: service-intentions/public-apiConfig entry deleted: service-intentions/frontendConfig entry deleted: service-intentions/nginxCreate Consul intentions for Hashicups and API GatewayConfig entry written: service-intentions/databaseConfig entry written: service-intentions/product-apiConfig entry written: service-intentions/payments-apiConfig entry written: service-intentions/public-apiConfig entry written: service-intentions/frontendConfig entry written: service-intentions/nginx

Verify API gateway deployment

Verify that the API gateway deployed successfully.

Use the nomad job command to retrieve information about the hashicups job.

$ nomad job allocs --namespace=ingress api-gatewayID        Node ID   Task Group   Version  Desired  Status   Created    Modifiedc1f36918  30b5f033  api-gateway  0        run      running  58m7s ago  57m46s ago

Use the consul catalog command to verify that the services are correctly registered inside Consul's catalog.

$ consul catalog servicesapi-gatewayconsulnomadnomad-client

Open the Nomad UI and log in with the ui -authenticate command. This command opens a web browser window on your machine. Alternatively, you can alternatively open the Nomad UI with the IP in Nomad_UI and log in with Nomad_UI_token.

$ nomad ui -authenticateOpening URL "https://18.116.52.247:4646" with one-time token

Nomad UI showing the jobs page. The page shows the API gateway job running.

In Nomad, go to the Jobs page and confirm that the api-gateway is running.

Then open the Consul UI and navigate to the Services page. The API gateway service is registered in the Consul service catalog with health checks.

Consul UI showing the services page. The page shows the API gatewat service running.

Navigate to the Intentions page. Verify that the service intentions are in place.

Consul UI showing the intentions page. The page shows the intentions necessary for HashiCups to work.

These intentions secure service-to-service communication between instances based on a service's identity.

Deploy HashiCups with service mesh

After you deploy the Consul API gateway, deploy the new version of the HashiCups job.

Run the HashiCups jobspec

Submit the HashiCups job to Nomad.

$ nomad job run 04.hashicups.nomad.hcl==> 2024-11-13T16:35:14+01:00: Monitoring evaluation "746b95fe"    2024-11-13T16:35:14+01:00: Evaluation triggered by job "hashicups"    2024-11-13T16:35:14+01:00: Allocation "04fffc73" created: node "b50f2376", group "product-api"    2024-11-13T16:35:14+01:00: Allocation "23d2a286" created: node "b50f2376", group "public-api"    2024-11-13T16:35:14+01:00: Allocation "444528dc" created: node "b50f2376", group "db"    2024-11-13T16:35:14+01:00: Allocation "5af579d5" created: node "7fb20437", group "payments"    2024-11-13T16:35:14+01:00: Allocation "8d4c039a" created: node "7fb20437", group "nginx"    2024-11-13T16:35:14+01:00: Allocation "ffddd50e" created: node "b50f2376", group "frontend"    2024-11-13T16:35:15+01:00: Evaluation within deployment: "5178e7b3"    2024-11-13T16:35:15+01:00: Evaluation status changed: "pending" -> "complete"==> 2024-11-13T16:35:15+01:00: Evaluation "746b95fe" finished with status "complete"==> 2024-11-13T16:35:15+01:00: Monitoring deployment "5178e7b3"  ✓ Deployment "5178e7b3" successful     2024-11-13T16:36:03+01:00    ID          = 5178e7b3    Job ID      = hashicups    Job Version = 0    Status      = successful    Description = Deployment completed successfully     Deployed    Task Group   Desired  Placed  Healthy  Unhealthy  Progress Deadline    db           1        1       1        0          2024-11-13T15:45:49Z    frontend     1        1       1        0          2024-11-13T15:45:54Z    nginx        1        1       1        0          2024-11-13T15:45:34Z    payments     1        1       1        0          2024-11-13T15:45:53Z    product-api  1        1       1        0          2024-11-13T15:45:53Z    public-api   1        1       1        0          2024-11-13T15:46:01Z

Verify the HashiCups deployment

Use the nomad job command to retrieve information about the hashicups job.

$ nomad job allocs hashicupsID        Node ID   Task Group   Version  Desired  Status   Created    Modified04fffc73  b50f2376  product-api  0        run      running  5m25s ago  4m45s ago23d2a286  b50f2376  public-api   0        run      running  5m25s ago  4m37s ago444528dc  b50f2376  db           0        run      running  5m25s ago  4m49s ago5af579d5  7fb20437  payments     0        run      running  5m25s ago  4m44s ago8d4c039a  7fb20437  nginx        0        run      running  5m25s ago  5m3s agoffddd50e  b50f2376  frontend     0        run      running  5m25s ago  4m44s ago

Use the consul catalog command to verify that the services are correctly registered inside Consul's catalog.

$ consul catalog servicesapi-gatewayconsuldatabasedatabase-sidecar-proxyfrontendfrontend-sidecar-proxynginxnginx-sidecar-proxynomadnomad-clientpayments-apipayments-api-sidecar-proxyproduct-apiproduct-api-sidecar-proxypublic-apipublic-api-sidecar-proxy

Open the Nomad UI and log in with the ui -authenticate command. The command will open a web browser window on your machine. Alternatively, open the Nomad UI with the IP in Nomad_UI and log in with Nomad_UI_token.

$ nomad ui -authenticateOpening URL "https://18.116.52.247:4646" with one-time token

There are now two jobs running on Nomad: api-gateway and hashicups.

Nomad UI showing the jobs page. The page shows both the API gateway and the HashiCups job running.

Navigate to the Topology page. The API gateway service is running on the public client node, making it accessible externally.

Nomad UI showing the topology page. The page shows the HashiCups and API gateway allocations.

Observe the changes in the UI, including allocations, when Nomad runs multiple jobs at the same time.

Then open the Consul UI and navigate to the Services page. Verify that each service is now registered in the Consul service catalog with health checks.

Consul UI showing the services page. The page shows HashiCups services with sidecar proxies running.

Additional icons describe node features, such as the existence of the service mesh proxy and the public ingress through the API gateway.

To view the Hashicups application, navigate to the public IP address of the API gateway.

In the following command, note the --namespace flag. The API gateway runs in a namespace that is separate from the application. You will need to trust the certificate in your browser.

$ nomad node status -verbose \    $(nomad job allocs --namespace=ingress api-gateway | grep -i running | awk '{print $2}') | \    grep -i public-ipv4 | awk -F "=" '{print $2}' | xargs | \    awk '{print "https://"$1":8443"}'

Output from the above command.

https://3.15.17.40:8443

Visit the address in your web browser to view the application.

HashiCups UI, running on the Nomad cluster.

Before you proceed, interact with the HashiCups application and then observe the changes in the Nomad and Consul UIs.

Cleanup

Stop the HashiCups deployment when you are ready to move on.

$ nomad job stop -purge hashicups==> 2024-11-04T17:26:49-05:00: Monitoring evaluation "8ff7d2ad"    2024-11-04T17:26:49-05:00: Evaluation triggered by job "hashicups"    2024-11-04T17:26:49-05:00: Evaluation status changed: "pending" -> "complete"==> 2024-11-04T17:26:49-05:00: Evaluation "8ff7d2ad" finished with status "complete"

Warning

Do not stop the API gateway job. You will use it in the next tutorial.

Stop the API gateway deployment when you are ready to move on.

$ nomad job stop --namespace ingress -purge api-gateway==> 2024-11-13T17:24:26+01:00: Monitoring evaluation "38867538"    2024-11-13T17:24:27+01:00: Evaluation triggered by job "api-gateway"    2024-11-13T17:24:27+01:00: Evaluation status changed: "pending" -> "complete"==> 2024-11-13T17:24:27+01:00: Evaluation "38867538" finished with status "complete"

Remove Consul intentions.

$ ./04.intentions.consul.sh -cleanConfigure environment.Clean previous configurations.Config entry deleted: service-intentions/databaseConfig entry deleted: service-intentions/product-apiConfig entry deleted: service-intentions/payments-apiConfig entry deleted: service-intentions/public-apiConfig entry deleted: service-intentions/frontendConfig entry deleted: service-intentions/nginxOnly cleaning selected...Exiting.

Remove Consul and Nomad configuration.

$ ./04.api-gateway.config.sh -cleanConfigure environment.Clean previous configurations.Config entry deleted: http-route/hashicups-http-routeConfig entry deleted: inline-certificate/api-gw-certicateConfig entry deleted: api-gateway/api-gatewayBinding rule "29fc7353-623c-ed89-5caa-b252ebc59aad" deleted successfullySuccessfully deleted namespace "ingress"!Auth method "nomad-workloads" deleted successfullyOnly cleaning selected...Exiting.

Next steps

In this tutorial, you deployed the Consul API gateway to secure access to your application. Then you deployed a version of HashiCups that uses Consul service mesh to fully secure services running on private client nodes.

For more information about to Consul API gateway integration with Nomad, refer to Consul ACL with Nomad Workload Identities and Deploy a Consul API Gateway on Nomad.

In the next tutorial, you will deploy the Nomad Autoscaler and use it to scale up the frontend service of HashiCups.

Integrate service discovery

Autoscale a service