Multi-cluster applications with Consul Enterprise admin partitions

30min
|
Enterprise
Consul
Video

Consul admin partitions give organizations the option to define and administer boundaries for services using Consul. This can help organizations managing services across teams and business units. Teams can benefit from managing and customizing their own Consul environment, without impacting other teams, or other Consul environments.

Some organizations want to allow organizational or business units to deploy their own installations of Consul Enterprise on their own Kubernetes clusters. Centrally managing multiple Consul installations can be an operational challenge for organizations. Instead of giving teams their own clusters, the organization can consolidate these installations onto a shared multi-tenant server cluster. This cluster serves as the control plane for Consul clients in the tenant clusters, and ensures separation between tenants of the system. This deployment model provides teams the autonomy to configure Consul and application networking as they require. This increases the team's flexibility to manage application deployments, and eliminates the operational overhead associated with managing individual server clusters.

In this tutorial, you will install and configure Consul Enterprise on two Kubernetes clusters with Helm. You will configure Admin Partitions and deploy a micro-services application called HashiCups. The services will be distributed across these two Kubernetes clusters using Consul admin partitions. A diagram showing this architecture is displayed below.

Picture of two AWS VPCs, containing two Kubernetes installlations, with two Consul Admin partitions, with services deployed to each partition. A description of these services is covered later in the tutorial

Prerequisites

An AWS account that is capable of deploying Amazon Elastic Kubernetes Service resources.
A Consul Enterprise license. Request a trial license on the Consul Enterprise trial registration page.

Warning

The sample repository is a Terraform project that deploys billable resources to your Amazon Web Services account. At minimum it will deploy 2 Amazon EKS Clusters and the supporting infrastructure for it, including at least one additional Classic Load Balancer. A cleanup script is provided at the end of this tutorial, but you are ultimately responsible for the resources deployed to your account.

Setting up the infrastructure

Clone GitHub repository

If using the sample repository, begin by cloning it locally, and change into the directory where the sample code is stored.

$ git clone https://github.com/hashicorp-education/learn-consul-kubernetes

Navigate into the repository folder.

$ cd learn-consul-kubernetes

Check out the git-tag associated with this tutorial.

$ git checkout v0.0.19

Navigate into the project folder for this tutorial.

$ cd consul-enterprise-admin-partitions-eks

Uploading the Consul Enterprise license

Start the tutorial by placing your Consul Enterprise license file in the consul_enterprise/ directory before deploying the infrastructure. Terraform uploads the license on your behalf. Ensure the file is named consul.hclic

Deploy the Infrastructure

Start by initializing the terraform project.

$ terraform init

Next, review the terraform plan.

$ terraform plan

You are now ready to deploy the infrastructure to AWS.

$ terraform apply -auto-approve

The infrastructure deployment takes approximately 30-40 minutes. You can use this time to review this tutorial, or watch the video below, demonstrating the usage and benefits of Consul admin partitions.

Deploying Kubernetes

Throughout the tutorial, you will complete tasks in context of Kubernetes and Consul Enterprise. To keep track, make note of the labels for each Consul and Kubernetes identifier:

Kubernetes cluster primary manages the Consul Enterprise server cluster.
Kubernetes cluster secondary manages the Consul Enterprise client cluster.

License upload verification

Verify the Consul Enterprise license secret named consul-ent-license exists on both clusters with the two following commands.

$ kubectl get secrets --context primary --namespace consulNAME                  TYPE                                  DATA   AGEconsul-ent-license    Opaque                                1      70mdefault-token-m2lbv   kubernetes.io/service-account-token   3      77m

$ kubectl get secrets --context secondary --namespace consulNAME                  TYPE                                  DATA   AGEconsul-ent-license    Opaque                                1      69mdefault-token-m2lbv   kubernetes.io/service-account-token   3      76m

Installing Consul Enterprise with Helm

Install Consul Enterprise with Helm by creating two values files for your Consul Enterprise installations. Navigate into the consul_enterprise folder.

$ cd consul_enterprise

For the Consul server cluster, create a values file named consul-values-server.yaml. For the Consul client cluster, create a values file named consul-values-client.yaml.

$ touch consul-values-server.yaml consul-values-client.yaml

Add the HashiCorp helm chart repository to your helm installation.

$ helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update

The configuration for the server and the client clusters are shown below. Copy and paste each tab's content into the values files you previously created for the server and the client.

These files are configured for the tutorial. In non-tutorial deployments, make note of the highlighted lines. Your organization may have specific guidance for you to follow regarding the values in the highlighted lines.

Consul Enterprise values files

global:  # If you are using the sample repository, do not change the global.name values in the consul-values files.  name: server  image: 'hashicorp/consul-enterprise:1.11.4-ent'  enableConsulNamespaces: true  # The name of the datacenter. Admin partitions are only supported in the same datacenter today.  datacenter: galaxy  enterpriseLicense:    # The Kubernetes secret key/value pair for your Consul Enterprise license. If using the terraform project provided with this tutorial    # this license is automatically uploaded for you as a secret if you place the license inside the `consul_enterprise` folder before deploying.    secretName: 'consul-ent-license'    secretKey: 'key'  # The stanza that sets up Admins Partitions and the reason you are reading this tutorial :)  adminPartitions:    enabled: true    service:      annotations: |        "service.beta.kubernetes.io/aws-load-balancer-scheme": "internal"  acls:    manageSystemACLs: true  logLevel: 'debug'  gossipEncryption:    autoGenerate: true  tls:    enableAutoEncrypt: true    enabled: true    verify: false    serverAdditionalDNSSANs:      # Subject Alternative Names that the TLS certificate will validate. In this case,      # these are wildcard values for Amazon EKS and Elastic Load Balancer services suppporting this tutorial.      # Note that wildcard values are not recommended for use in production environments.      - '*.us-east-1.elb.amazonaws.com'      - '*.gr7.us-east-1.eks.amazonaws.com'server:  exposeGossipAndRpcPorts: true  replicas: 1ui:  enabled: trueconnectInject:  aclBindingRuleSelector: ''  consulNamepaces:    mirroringK8S: true  transparentProxy:    defaultEnabled: false  # inject an envoy sidecar into every new pod, except for those with annotations that prevent injection  enabled: true  # these settings enable L7 metrics collection and are new in 1.5  centralConfig:    enabled: true    # proxyDefaults is a raw json string that will be applied to all Connect    # proxy sidecar pods that can include any valid configuration for the    # configured proxy.    # proxyDefaults: |    #   {    #     "envoy_prometheus_bind_addr": "0.0.0.0:9102"    #   }controller:  enabled: true

global:  name: client  enabled: false  image: hashicorp/consul-enterprise:1.11.4-ent  enableConsulNamespaces: true  # Must match consul-values-server.yaml  datacenter: galaxy  enterpriseLicense:    secretName: 'consul-ent-license'    secretKey: 'key'  adminPartitions:    enabled: true    # The name of the admin partition for the client partition. This can be any value. An admin partition typically represents a single Consul Enterprise cluster on one Kubernetes cluster.    name: 'tereknor'  acls:    manageSystemACLs: true    bootstrapToken:      # Copy from server      secretName: server-partitions-acl-token      secretKey: token  gossipEncryption:    # copy from server    secretName: server-gossip-encryption-key    secretKey: key  tls:    verify: false    enabled: true    enableAutoEncrypt: true    caKey:      # copy from server      secretName: server-ca-key      secretKey: tls.key    caCert:      # copy from server      secretName: server-ca-cert      secretKey: tls.crtexternalServers:  enabled: true  hosts:    # The hostname of the admin-partition `LoadBalancer` service Running on the server cluster    - YourValueGoesHere  useSystemRoots: false  # The client cluster's hostname for the Kubernetes control plane. Obtained by running `kubectl --cluster-info` for that cluster.  k8sAuthMethodHost: YourValueGoesHereclient:  enabled: true  join:    # The hostname of the admin-partition `LoadBalancer` service Running on the server cluster    - YourValueGoesHereconnectInject:  aclBindingRuleSelector: ''  enabled: true  transparentProxy:    defaultEnabled: false  consulNamespaces:    mirroringK8S: truecontroller:  enabled: true

When admin partitions are enabled, the Consul server cluster generates a partition for itself, named default. For Consul client clusters which register themselves to the Consul server cluster, these partition names are configurable in the global.adminPartitions stanza of the cluster's values file. For this tutorial, refer to line 14 of the consul-values-client.yaml values file shown above.

To learn more about Consul on Kubernetes, visit the Consul on Kubernetes repository.

Consul Enterprise Server

For the server Consul installation, install Consul Enterprise on the primary Kubernetes cluster.

$ kubectl config use-context primary && helm install --wait hashicorp-server hashicorp/consul --namespace consul --version "0.43.0" --values consul-values-server.yaml   Switched to context "primary".   NAME: hashicorp-server  LAST DEPLOYED: Mon Nov 29 14:35:20 2021  NAMESPACE: consul  STATUS: deployed  REVISION: 1  NOTES:  Thank you for installing HashiCorp Consul!   Now that you have deployed Consul, you should look over the docs on using  Consul with Kubernetes available here:   https://www.consul.io/docs/platform/k8s/index.html    Your release is named hashicorp-server.   To learn more about the release, run:     $ helm status hashicorp-server    $ helm get all hashicorp-server

Confirm the Consul primary cluster has admin partitions enabled, and includes a member named server-server-0. Use the consul members command inside the server-server-0 pod.

$ kubectl exec -it server-server-0 --namespace consul consul membersNode                          Address            Status  Type    Build       Protocol  DC      Partition  Segmentserver-server-0               10.100.2.57:8301   alive   server  1.11.4+ent  2         galaxy  default    <all>ip-10-100-1-218.ec2.internal  10.100.1.238:8301  alive   client  1.11.4+ent  2         galaxy  default    <default>ip-10-100-1-43.ec2.internal   10.100.1.125:8301  alive   client  1.11.4+ent  2         galaxy  default    <default>ip-10-100-2-148.ec2.internal  10.100.2.93:8301   alive   client  1.11.4+ent  2         galaxy  default    <default>ip-10-100-2-44.ec2.internal   10.100.2.135:8301  alive   client  1.11.4+ent  2         galaxy  default    <default>

Observe the existence of the default partition name in the Partition column. The partition for the Consul server cluster is always named default.

Installing the Consul Enterprise client cluster.

Before installing Consul Enterprise on the Kubernetes secondary cluster, configure the Consul client installation to communicate with two following services:

The Kubernetes primary cluster's admin partition service, created by Consul.
The Kubernetes control plane hosted on its own cluster.

Confirm the Kubernetes primary cluster has an active service named consul-partition-service with a value in the EXTERNAL-IP column. This service is created by Consul during the Helm installation. This value is a hostname or IP address. A Consul client uses this hostname to access and register its admin partition with the Consul Enterprise server cluster. Copy this hostname. A shortcut to obtain the EXTERNAL-IP is provided below.

Note

Admin Partitions, when enabled, creates a Load Balancer for itself. In AWS, this creates a new Classic Load Balancer, a billable resource.

$ kubectl get services --context primary --selector="app=consul,component=server" --namespace consul --output jsonpath="{range .items[*]}{@.status.loadBalancer.ingress[*].hostname}{end}"a781098c658b44fa79d145d7a4eb192d-1813301156.us-east-1.elb.amazonaws.com

Edit the file consul-values-client.yaml, adding the hostname as a value for the client.join and externalServers.host keys.

Next, copy the control-plane hostname for the secondary Kubernetes cluster. A shortcut is provided below.

$ TERM=dumb kubectl --context secondary cluster-info | awk '/Kubernetes control plane/ {print $NF}'

The prefix to kubectl, TERM=dumb, removes the stylization of text that cluster-info outputs when returning the hostname. This can cause issues when copying and pasting text, when the stylization is present.

Update the externalServers.k8sAuthMethodHost key with the control-plane hostname of the secondary Kubernetes cluster.

Below, observe the keys for each value where to update the consul-values-client.yaml file.

externalServers:  enabled: true  hosts:    # The hostname of the admin-partition `LoadBalancer` service Running on the server cluster    - YourValueGoesHere  useSystemRoots: false  # The client cluster's hostname for the Kubernetes control plane. Obtained by running `kubectl --cluster-info` for that cluster.  k8sAuthMethodHost: YourValueGoesHereclient:  enabled: true  join:    # The hostname of the admin-partition `LoadBalancer` service Running on the server cluster    - YourValueGoesHere

Secrets Management for the Consul client cluster

For the Consul client installation, configure the Consul Enterprise clusters to use HTTPS. The Consul server cluster is already configured to use HTTPS. However, the Consul client cluster requires you to perform a few actions before enabling HTTPS.

Copy the HTTPS CA certifcate and HTTPS CA Key from the primary Kubernetes cluster to the secondary Kubernetes cluster.

$ kubectl get --namespace consul secret server-ca-cert --context primary -o yaml | kubectl apply --context secondary --namespace consul --filename - && kubectl get --namespace consul secret server-ca-key --context primary -o yaml  | kubectl apply --namespace consul --context secondary --filename -   secret/server-ca-cert created  secret/server-ca-key created

The Consul Enterprise client cluster needs additional secrets generated by the Consul Enterprise server installation. These secrets let the client authenticate, and communicate with the Consul server cluster. You will need access control list (ACL) tokens for the admin partition, client services, and for the initial bootstrap of the Consul client cluster. You will also need the gossip encryption key required to use gossip encryption on the Consul client cluster.

Use the commands below to copy the secrets from the primary Kubernetes cluster to the secondary Kubernetes cluster.

Partition Token

$  kubectl get --namespace consul secret server-partitions-acl-token --context primary -o yaml   | kubectl apply --namespace consul --context secondary --filename - secret/server-partitions-acl-token created

Client ACL token

$  kubectl get --namespace consul secret server-client-acl-token --context primary -o yaml      | kubectl apply --namespace consul --context secondary --filename - secret/server-client-acl-token created

Bootstrap ACL Token

$  kubectl get --namespace consul secret server-bootstrap-acl-token --context primary -o yaml   | kubectl apply --namespace consul --context secondary --filename - secret/server-bootstrap-acl-token created

Gossip Encryption Key

$  kubectl get --namespace consul secret server-gossip-encryption-key --context primary -o yaml | kubectl apply --namespace consul --context secondary --filename - secret/server-gossip-encryption-key created

Confirm the secrets copied from the primary Kubernetes cluster, are present on the secondary Kubernetes cluster.

$ kubectl get secrets --namespace consul --context secondary NAME                           TYPE                                  DATA   AGEconsul-ent-license             Opaque                                1      69mdefault-token-brwk8            kubernetes.io/service-account-token   3      77mserver-bootstrap-acl-token     Opaque                                1      60sserver-ca-cert                 Opaque                                1      100sserver-ca-key                  Opaque                                1      98sserver-client-acl-token        Opaque                                1      68sserver-gossip-encryption-key   Opaque                                1      52sserver-partitions-acl-token    Opaque                                1      79s

The secondary Kubernetes cluster is prepared for Consul client cluster installation. Deploy theconsul-values-client.yaml chart with Helm.

$ kubectl config use-context secondary && helm install --wait hashicorp-client hashicorp/consul --namespace consul --version "0.43.0" --values consul-values-client.yaml   Switched to context "secondary".   NAME: hashicorp-client  LAST DEPLOYED: Sat Jan 15 17:48:02 2022  NAMESPACE: consul  STATUS: deployed  REVISION: 1  NOTES:  Thank you for installing HashiCorp Consul!   Now that you have deployed Consul, you should look over the docs on using  Consul with Kubernetes available here:   https://www.consul.io/docs/platform/k8s/index.html    Your release is named hashicorp-client.   To learn more about the release, run:     $ helm status hashicorp-client    $ helm get all hashicorp-client

Confirm the Consul Enterprise services and pods on the secondary Kubernetes cluster have been deployed successfully. Issue the commands below in your environment for verification.

$ kubectl get svc --namespace consul --context secondaryNAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGEclient-connect-injector-svc   ClusterIP   10.100.63.19    <none>        443/TCP   4m49sclient-controller-webhook     ClusterIP   10.100.106.50   <none>        443/TCP   4m49s

Note the difference from the output of this command in comparison to previously inputting the same command on the Consul primary cluster. There is no server-server-0 pod running on this Kubernetes cluster, nor a consul-partition-service service.

Verify the pods for Consul Enterprise are present and active on the Kubernetes secondary cluster.

$ kubectl get pods --namespace consul --context secondaryNAME                                           READY   STATUS    RESTARTS   AGEclient-client-4tmnx                            1/1     Running   0          4m6sclient-client-5gf6q                            1/1     Running   0          4m6sclient-client-bvllz                            1/1     Running   0          4m6sclient-client-vgght                            1/1     Running   0          4m6sclient-connect-injector-74f86c67c7-54cdq       1/1     Running   0          4m6sclient-connect-injector-74f86c67c7-zlxng       1/1     Running   0          4m6sclient-controller-59457d56f4-rdqkh             1/1     Running   0          4m6sclient-webhook-cert-manager-648668664d-5lq7s   1/1     Running   0          4m6s

Confirming the client cluster's admin partition.

When you deployed the file consul-values-client.yaml, a partition for the Consul client cluster was created, named tereknor. Confirm this admin partition is present on the Consul client cluster and is registered to the Consul server cluster.

From the output of the kubectl get pods --context secondary command above, use one of the client pods affixed with random characters on the end, such as client-client-4tmnx, to verify the members in the Consul client cluster.

$ kubectl exec --namespace consul -it --context secondary -c consul client-client-4tmnx consul members Node                          Address            Status  Type    Build       Protocol  DC      Partition  Segmentserver-server-0               10.100.2.57:8301   alive   server  1.11.4+ent  2         galaxy  default    <all>ip-172-30-1-118.ec2.internal  172.30.1.31:8301   alive   client  1.11.4+ent  2         galaxy  tereknorip-172-30-1-241.ec2.internal  172.30.1.30:8301   alive   client  1.11.4+ent  2         galaxy  tereknorip-172-30-2-241.ec2.internal  172.30.2.108:8301  alive   client  1.11.4+ent  2         galaxy  tereknorip-172-30-2-33.ec2.internal   172.30.2.121:8301  alive   client  1.11.4+ent  2         galaxy  tereknor

Notice that the server-server-0 node from the Consul Enterprise server cluster is listed as the server for the client cluster, in the type column. This confirms that your Consul client cluster is registered to the Consul server cluster.

For additional verification, check the logs on the same pod, with kubectl.

$ kubectl logs --namespace consul --context secondary client-client-4tmnx -c consul 2022-01-15T23:48:33.297Z [INFO]  agent.client.serf.lan: serf: EventMemberJoin: server-server-0 10.100.2.57

To further verify the partition is registered to the Consul server cluster, use consul partition list. This command returns all partitions the Consul server cluster is aware of, and their descriptions. This is an authenticated command. You will use the ACL token from the primary cluster to authenticate, and pass the value as an argument to the -token flag in consul partition list.

Begin by setting the token in a shell variable. The decoded value in your shell output will be unique in comparison to this tutorial's output below.

$ ACL_TOKEN=$(kubectl get secret server-bootstrap-acl-token --namespace consul --context primary --template '{{ .data.token | base64decode }}') && echo "${ACL_TOKEN}" 2a5f0240-8be6-98c3-5c37-9db4723f67ad

Use the variable $ACL_TOKEN as the argument for the -token flag.

$ kubectl exec -it --namespace consul --context primary server-server-0 -- consul partition list -token "${ACL_TOKEN}"  default:    Description:        Builtin Default Partition  tereknor:    Description:        Created by Helm installation

To observe the members of a specific partition, use consul catalog nodes with the --partition flag, using the $ACL_TOKEN to use this authenticated command.

$ kubectl exec -it --namespace consul --context primary server-server-0 -- consul catalog nodes --partition tereknor -token "${ACL_TOKEN}"   Node                         ID        Address      Partition  DC  ip-172-30-1-8.ec2.internal   8c54f491  172.30.1.8   tereknor   galaxy  ip-172-30-1-84.ec2.internal  aded415b  172.30.1.84  tereknor   galaxy  ip-172-30-2-45.ec2.internal  3a9077c0  172.30.2.45  tereknor   galaxy  ip-172-30-2-91.ec2.internal  efef2194  172.30.2.91  tereknor   galaxy

Deploy HashiCups as a distributed application

To deploy an application to your cluster, you will use HashiCups, a micro-services architecture. HashiCups serves as a shopping cart for HashiCorp-themed beverages. It has specific permissions for service communication: Its database communicates with a product API service. The front end communicates with a public API service. The public api brokers communication between the product api and the payments service.

Picture of the HashiCups architecture, as described in the previous paragraph. A picture of a user with flowchart arrows indicating communication: A User connects from the Public. The Frontend communicates over mTLS with the Public API. The Public API communicates over mTLS with the Payments service and the Product API. The Product API communicates over mTLS with the postgres database.

In this tutorial, the postgres database resides on the primary Kubernetes cluster, belonging to the Consul server cluster. This is a common database that other products can connect to, not just HashiCups. The remaining services are deployed to the secondary Kubernetes cluster, belonging to the client Consul Enterprise cluster.

With both Kubernetes clusters, any pod deployed will be bootstrapped with a consul agent to each container. When resources are deployed, they are assigned to the partition of that cluster. For the Consul server cluster, resources will belong to the default partition. For Consul client clusters, resources will belong to the tereknor partition.

Exporting services with Custom Resource Definitions

In your installation, Consul Enterprise is configured with Access Control Lists enabled. With ACLs enabled, you will export the services that communicate across partitions, namely the database service, and the products API. You will use Consul Intentions to grant services access to these cross-cluster services.

Note

Learn about Access Control Lists and Consul Intentions by visiting the Access Control Lists Learn Tutorial and the Application Aware Intentions with Consul Service Mesh Learn Tutorial

Primary Cluster

$ kubectl apply --filename hashicups/crds/crd-exported-service-primary.yaml --filename hashicups/crds/crd-intentions.yaml --context primary   exportedservices.consul.hashicorp.com/default created  serviceintentions.consul.hashicorp.com/public-api created  serviceintentions.consul.hashicorp.com/products-api created  serviceintentions.consul.hashicorp.com/payments created  serviceintentions.consul.hashicorp.com/postgres created

Secondary Cluster

$ kubectl apply --filename hashicups/crds/crd-exported-service-secondary.yaml --filename hashicups/crds/crd-intentions.yaml --context secondary   exportedservices.consul.hashicorp.com/tereknor created  serviceintentions.consul.hashicorp.com/public-api created  serviceintentions.consul.hashicorp.com/products-api created  serviceintentions.consul.hashicorp.com/payments created  serviceintentions.consul.hashicorp.com/postgres created

Deploying HashiCups across clusters

The following code block groups create a tab for each service by their kind value in the Kubernetes plan file. To deploy the services, a final tab Complete is available in each code block group, containing all the of the services in one single continuous block. The deployment instructions reflect deploying the contents of this tab with a single file.

Primary Cluster

The database will be deployed inside Consul server cluster's default partition, on the primary Kubernetes cluster. Consul assigns the services to their partition automatically.

Database

Create a file called postgres.yaml and paste the contents of the "Complete" tab inside.

$ touch postgres.yaml

Service: Database

---apiVersion: v1kind: Servicemetadata:  name: postgres  labels:    app: postgresspec:  type: ClusterIP  ports:    - port: 5432      targetPort: 5432  selector:    app: postgres

---apiVersion: apps/v1kind: Deploymentmetadata:  name: postgresspec:  replicas: 1  selector:    matchLabels:      service: postgres      app: postgres  template:    metadata:      labels:        service: postgres        app: postgres      annotations:        prometheus.io/scrape: 'true'        prometheus.io/port: '9102'        consul.hashicorp.com/connect-inject: 'true'    spec:      serviceAccountName: postgres      containers:        - name: postgres          image: hashicorpdemoapp/product-api-db:v0.0.13          ports:            - containerPort: 5432          env:            - name: POSTGRES_DB              value: products            - name: POSTGRES_USER              value: postgres            - name: POSTGRES_PASSWORD              value: password          # only listen on loopback so only access is via connect proxy          args: ['-c', 'listen_addresses=127.0.0.1']          volumeMounts:            - mountPath: '/var/lib/postgresql/data'              name: 'pgdata'      volumes:        - name: pgdata          emptyDir: {}

---apiVersion: v1kind: Servicemetadata:  name: postgres  labels:    app: postgresspec:  type: ClusterIP  ports:    - port: 5432      targetPort: 5432  selector:    app: postgres ---apiVersion: consul.hashicorp.com/v1alpha1kind: ServiceDefaultsmetadata:  name: postgresspec:  protocol: 'tcp'---apiVersion: v1kind: ServiceAccountmetadata:  name: postgresautomountServiceAccountToken: true ---apiVersion: apps/v1kind: Deploymentmetadata:  name: postgresspec:  replicas: 1  selector:    matchLabels:      service: postgres      app: postgres  template:    metadata:      labels:        service: postgres        app: postgres      annotations:        prometheus.io/scrape: 'true'        prometheus.io/port: '9102'        consul.hashicorp.com/connect-inject: 'true'    spec:      serviceAccountName: postgres      containers:        - name: postgres          image: hashicorpdemoapp/product-api-db:v0.0.13          ports:            - containerPort: 5432          env:            - name: POSTGRES_DB              value: products            - name: POSTGRES_USER              value: postgres            - name: POSTGRES_PASSWORD              value: password          # only listen on loopback so only access is via connect proxy          args: ['-c', 'listen_addresses=127.0.0.1']          volumeMounts:            - mountPath: '/var/lib/postgresql/data'              name: 'pgdata'      volumes:        - name: pgdata          emptyDir: {}

Deploy the database service:

$ kubectl apply --context primary --filename postgres.yaml   service/postgres created  servicedefaults.consul.hashicorp.com/postgres created  serviceaccount/postgres created  deployment.apps/postgres created

Secondary Cluster

You will deploy the remaining HashiCups micro-services to the Consul client cluster's tereknor partition, on the secondary Kubernetes cluster. Consul assigns the services to their partition automatically.

Products API

With the Products API deployment, note the db_connection string in the data.config stanza in the ConfigMap plan file. The database is deployed to the primary Kubernetes cluster in a separate network, but Consul Enterprise makes this service available to the Kubernetes secondary cluster via the loopback address of 127.0.0.1, via the Envoy Proxy. Learn about the Envoy proxy in the Load Balancing Services in Consul Service Mesh with Envoy Learn Tutorial.

Create a file called products-api.yaml and paste the contents of the "Complete" tab inside.

$ touch products-api.yaml

Service: Products API

---apiVersion: v1kind: Servicemetadata:  name: products-api  labels:    app: products-apispec:  selector:    app: products-api  ports:    - name: http      protocol: TCP      port: 9090      targetPort: 9090

apiVersion: v1kind: ConfigMapmetadata:  name: db-configmapdata:  config: |    {      "db_connection": "host=127.0.0.1 port=5432 user=postgres password=password dbname=products sslmode=disable",      "bind_address": ":9090",      "metrics_address": ":9103"    }

---apiVersion: apps/v1kind: Deploymentmetadata:  name: products-api  labels:    app: products-apispec:  replicas: 1  selector:    matchLabels:      app: products-api  template:    metadata:      labels:        app: products-api      annotations:        prometheus.io/scrape: 'true'        prometheus.io/port: '9102'        consul.hashicorp.com/connect-inject: 'true'        # This value is the name of the service, the name of the namespace, and the name of the partition in that order.        consul.hashicorp.com/connect-service-upstreams: 'postgres.default.default:5432'    spec:      serviceAccountName: products-api      volumes:        - name: config          configMap:            name: db-configmap            items:              - key: config                path: conf.json      containers:        - name: products-api          image: hashicorpdemoapp/product-api:v0.0.19          imagePullPolicy: Always          ports:            - containerPort: 9090            - containerPort: 9103          env:            - name: 'CONFIG_FILE'              value: '/config/conf.json'          livenessProbe:            httpGet:              path: /health              port: 9090            initialDelaySeconds: 15            timeoutSeconds: 1            periodSeconds: 10            failureThreshold: 30          volumeMounts:            - name: config              mountPath: /config              readOnly: true

---apiVersion: v1kind: Servicemetadata:  name: products-api  # new  labels:    app: products-apispec:  selector:    app: products-api  ports:    - name: http      protocol: TCP      port: 9090      targetPort: 9090---apiVersion: v1kind: ServiceAccountmetadata:  name: products-apiautomountServiceAccountToken: true---apiVersion: consul.hashicorp.com/v1alpha1kind: ServiceDefaultsmetadata:  name: products-apispec:  protocol: 'tcp'---apiVersion: v1kind: ConfigMapmetadata:  name: db-configmapdata:  config: |    {      "db_connection": "host=127.0.0.1 port=5432 user=postgres password=password dbname=products sslmode=disable",      "bind_address": ":9090",      "metrics_address": ":9103"    }---apiVersion: apps/v1kind: Deploymentmetadata:  name: products-api  labels:    app: products-apispec:  replicas: 1  selector:    matchLabels:      app: products-api  template:    metadata:      labels:        app: products-api      annotations:        prometheus.io/scrape: 'true'        prometheus.io/port: '9102'        consul.hashicorp.com/connect-inject: 'true'        # This value is the name of the service, the name of the namespace, and the name of the partition in that order.        consul.hashicorp.com/connect-service-upstreams: 'postgres.default.default:5432'    spec:      serviceAccountName: products-api      volumes:        - name: config          configMap:            name: db-configmap            items:              - key: config                path: conf.json      containers:        - name: products-api          image: hashicorpdemoapp/product-api:v0.0.19          imagePullPolicy: Always          ports:            - containerPort: 9090            - containerPort: 9103          env:            - name: 'CONFIG_FILE'              value: '/config/conf.json'          livenessProbe:            httpGet:              path: /health              port: 9090            initialDelaySeconds: 15            timeoutSeconds: 1            periodSeconds: 10            failureThreshold: 30          volumeMounts:            - name: config              mountPath: /config              readOnly: true

Deploy the Products API service:

$ kubectl apply --context secondary --filename products-api.yaml service/products-api createdserviceaccount/products-api createdservicedefaults.consul.hashicorp.com/products-api createdconfigmap/db-configmap createddeployment.apps/products-api created

Public API

Create a file called public-api.yaml and paste the contents of the "Complete" tab inside.

$ touch public-api.yaml

Service: Public API

---apiVersion: v1kind: Servicemetadata:  name: public-api  labels:    app: public-apispec:  type: ClusterIP  ports:    - port: 8080      targetPort: 8080  selector:    app: public-api

---apiVersion: apps/v1kind: Deploymentmetadata:  name: public-apispec:  replicas: 1  selector:    matchLabels:      service: public-api      app: public-api  template:    metadata:      labels:        service: public-api        app: public-api      annotations:        prometheus.io/scrape: 'true'        prometheus.io/port: '9102'        consul.hashicorp.com/connect-inject: 'true'        consul.hashicorp.com/connect-service-upstreams: 'products-api:9090, payments:1800'    spec:      serviceAccountName: public-api      containers:        - name: public-api          image: hashicorpdemoapp/public-api:v0.0.5          imagePullPolicy: Always          ports:            - containerPort: 8080          env:            - name: BIND_ADDRESS              value: ':8080'            - name: PRODUCT_API_URI              value: 'http://localhost:9090'            - name: PAYMENT_API_URI              value: 'http://localhost:1800'        - name: jaeger-agent          image: jaegertracing/jaeger-agent:latest          imagePullPolicy: IfNotPresent          ports:            - containerPort: 5775              name: zk-compact-trft              protocol: UDP            - containerPort: 5778              name: config-rest              protocol: TCP            - containerPort: 6831              name: jg-compact-trft              protocol: UDP            - containerPort: 6832              name: jg-binary-trft              protocol: UDP            - containerPort: 14271              name: admin-http              protocol: TCP          args:            - --reporter.grpc.host-port=dns:///jaeger-collector-headless.default:14250            - --reporter.type=grpc

---apiVersion: v1kind: Servicemetadata:  name: public-api  labels:    app: public-apispec:  type: ClusterIP  ports:    - port: 8080      targetPort: 8080  selector:    app: public-api---apiVersion: v1kind: ServiceAccountmetadata:  name: public-apiautomountServiceAccountToken: true---apiVersion: apps/v1kind: Deploymentmetadata:  name: public-apispec:  replicas: 1  selector:    matchLabels:      service: public-api      app: public-api  template:    metadata:      labels:        service: public-api        app: public-api      annotations:        prometheus.io/scrape: 'true'        prometheus.io/port: '9102'        consul.hashicorp.com/connect-inject: 'true'        consul.hashicorp.com/connect-service-upstreams: 'products-api:9090, payments:1800'    spec:      serviceAccountName: public-api      containers:        - name: public-api          image: hashicorpdemoapp/public-api:v0.0.5          imagePullPolicy: Always          ports:            - containerPort: 8080          env:            - name: BIND_ADDRESS              value: ':8080'            - name: PRODUCT_API_URI              value: 'http://localhost:9090'            - name: PAYMENT_API_URI              value: 'http://localhost:1800'        - name: jaeger-agent          image: jaegertracing/jaeger-agent:latest          imagePullPolicy: IfNotPresent          ports:            - containerPort: 5775              name: zk-compact-trft              protocol: UDP            - containerPort: 5778              name: config-rest              protocol: TCP            - containerPort: 6831              name: jg-compact-trft              protocol: UDP            - containerPort: 6832              name: jg-binary-trft              protocol: UDP            - containerPort: 14271              name: admin-http              protocol: TCP          args:            - --reporter.grpc.host-port=dns:///jaeger-collector-headless.default:14250            - --reporter.type=grpc

Deploy the Public API service:

$ kubectl apply --context secondary --filename public-api.yaml service/public-api createdserviceaccount/public-api createddeployment.apps/public-api created

Payments

Create a file called payments.yaml and paste the contents of the "Complete" tab inside.

$ touch payments.yaml

Service: Payments API

---apiVersion: v1kind: Servicemetadata:  name: payments  labels:    app: paymentsspec:  selector:    app: payments  ports:    - name: http      protocol: TCP      port: 1800      targetPort: 8080

---apiVersion: apps/v1kind: Deploymentmetadata:  name: payments  labels:    app: paymentsspec:  replicas: 1  selector:    matchLabels:      app: payments  template:    metadata:      labels:        app: payments      annotations:        consul.hashicorp.com/connect-inject: 'true'    spec:      serviceAccountName: payments      containers:        - name: payments          image: hashicorpdemoapp/payments:v0.0.12          imagePullPolicy: Always          ports:            - containerPort: 8080

---apiVersion: v1kind: Servicemetadata:  name: payments  labels:    app: paymentsspec:  selector:    app: payments  ports:    - name: http      protocol: TCP      port: 1800      targetPort: 8080---apiVersion: v1kind: ServiceAccountmetadata:  name: paymentsautomountServiceAccountToken: true---apiVersion: consul.hashicorp.com/v1alpha1kind: ServiceDefaultsmetadata:  name: paymentsspec:  protocol: 'http'---apiVersion: apps/v1kind: Deploymentmetadata:  name: payments  labels:    app: paymentsspec:  replicas: 1  selector:    matchLabels:      app: payments  template:    metadata:      labels:        app: payments      annotations:        consul.hashicorp.com/connect-inject: 'true'    spec:      serviceAccountName: payments      containers:        - name: payments          image: hashicorpdemoapp/payments:v0.0.12          imagePullPolicy: Always          ports:            - containerPort: 8080

Deploy the Payments service:

$ kubectl apply --context secondary --filename payments.yaml service/payments createdserviceaccount/payments createdservicedefaults.consul.hashicorp.com/payments createddeployment.apps/payments created

Frontend

Create a file called frontend.yaml and paste the contents of the "Complete" tab inside.

$ touch frontend.yaml

Service: Frontend

apiVersion: v1kind: Servicemetadata:  name: frontend  labels:    app: frontendspec:  type: LoadBalancer  ports:    - port: 80      targetPort: 80  selector:    app: frontend

apiVersion: v1kind: ConfigMapmetadata:  name: nginx-configmapdata:  config: |    # /etc/nginx/conf.d/default.conf    server {        listen       80;        server_name  localhost;         #charset koi8-r;        #access_log  /var/log/nginx/host.access.log  main;         location / {            root   /usr/share/nginx/html;            index  index.html index.htm;        }         # Proxy pass the api location to save CORS        # Use location exposed by Consul connect        location /api {            proxy_pass http://localhost:8080;            proxy_http_version 1.1;            proxy_set_header Upgrade $http_upgrade;            proxy_set_header Connection "Upgrade";            proxy_set_header Host $host;        }         error_page   500 502 503 504  /50x.html;        location = /50x.html {            root   /usr/share/nginx/html;        }    }

---apiVersion: apps/v1kind: Deploymentmetadata:  name: frontendspec:  replicas: 1  selector:    matchLabels:      service: frontend      app: frontend  template:    metadata:      labels:        service: frontend        app: frontend      annotations:        prometheus.io/scrape: 'true'        prometheus.io/port: '9102'        consul.hashicorp.com/connect-inject: 'true'        consul.hashicorp.com/connect-service-upstreams: 'public-api:8080'    spec:      serviceAccountName: frontend      volumes:        - name: config          configMap:            name: nginx-configmap            items:              - key: config                path: default.conf      containers:        - name: frontend          image: hashicorpdemoapp/frontend:v0.0.7          imagePullPolicy: Always          ports:            - containerPort: 80          volumeMounts:            - name: config              mountPath: /etc/nginx/conf.d              readOnly: true

apiVersion: v1kind: Servicemetadata:  name: frontend  labels:    app: frontendspec:  type: LoadBalancer  ports:    - port: 80      targetPort: 80  selector:    app: frontend---apiVersion: v1kind: ServiceAccountmetadata:  name: frontendautomountServiceAccountToken: true---apiVersion: consul.hashicorp.com/v1alpha1kind: ServiceDefaultsmetadata:  name: frontendspec:  protocol: http---apiVersion: v1kind: ConfigMapmetadata:  name: nginx-configmapdata:  config: |    # /etc/nginx/conf.d/default.conf    server {        listen       80;        server_name  localhost;         #charset koi8-r;        #access_log  /var/log/nginx/host.access.log  main;         location / {            root   /usr/share/nginx/html;            index  index.html index.htm;        }         # Proxy pass the api location to save CORS        # Use location exposed by Consul connect        location /api {            proxy_pass http://localhost:8080;            proxy_http_version 1.1;            proxy_set_header Upgrade $http_upgrade;            proxy_set_header Connection "Upgrade";            proxy_set_header Host $host;        }         error_page   500 502 503 504  /50x.html;        location = /50x.html {            root   /usr/share/nginx/html;        }    } ---apiVersion: apps/v1kind: Deploymentmetadata:  name: frontendspec:  replicas: 1  selector:    matchLabels:      service: frontend      app: frontend  template:    metadata:      labels:        service: frontend        app: frontend      annotations:        prometheus.io/scrape: 'true'        prometheus.io/port: '9102'        consul.hashicorp.com/connect-inject: 'true'        consul.hashicorp.com/connect-service-upstreams: 'public-api:8080'    spec:      serviceAccountName: frontend      volumes:        - name: config          configMap:            name: nginx-configmap            items:              - key: config                path: default.conf      containers:        - name: frontend          image: hashicorpdemoapp/frontend:v0.0.7          imagePullPolicy: Always          ports:            - containerPort: 80          volumeMounts:            - name: config              mountPath: /etc/nginx/conf.d              readOnly: true

Deploy the Frontend service:

$ kubectl apply --context secondary --filename frontend.yaml service/frontend createdserviceaccount/frontend createdservicedefaults.consul.hashicorp.com/frontend createdconfigmap/nginx-configmap createddeployment.apps/frontend created

Validation steps

Next, verify the HashiCups deployment is active.

HashiCups services are deployed

Observe the pods for each Kubernetes plan file you deployed are in a ready state according to the READY column. Each pod is ready when all defined pods are available. For example, the public-api is defined with 3 pods. In the output below, 3 of 3 pods are ready.

$ kubectl get pods --namespace consul --context secondary NAME                                                         READY   STATUS    RESTARTS   AGEclient-client-4tmnx                            1/1     Running           0          34mclient-client-5gf6q                            1/1     Running           0          34mclient-client-bvllz                            1/1     Running           0          34mclient-client-vgght                            1/1     Running           0          34mclient-connect-injector-74f86c67c7-54cdq       1/1     Running           0          34mclient-connect-injector-74f86c67c7-zlxng       1/1     Running           0          34mclient-controller-59457d56f4-rdqkh             1/1     Running           0          34mclient-webhook-cert-manager-648668664d-5lq7s   1/1     Running           0          34m

$ kubectl get pods --context secondary NAME                                                         READY   STATUS    RESTARTS   AGEfrontend-86bff5b5b6-jn6sm                                    2/2     Running   0          40spayments-7564d7bfb5-26qdq                                    2/2     Running   0          3m27sproducts-api-7bbd9458d7-tfvhf                                2/2     Running   5          8m48spublic-api-77b4665585-9lvfv                                  3/3     Running   0          6m8s

HashiCups frontend is reachable

Verify the frontend service has a value in the EXTERNAL-IP column. Copy the EXTERNAL-IP value for the frontend service, and paste it into your web browser.

DNS propagation for the HashiCups application can take a few minutes on AWS, so you may initially observe NXDOMAIN errors returned by DNS until the DNS Time To Live (TTL) refreshes its understanding of services associated with that hostname.

$ kubectl get services --context secondaryNAME                          TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)        AGEfrontend                      LoadBalancer   10.100.17.104    a14e155508a234ced82ba80c50baa6de-2091774586.us-east-1.elb.amazonaws.com   80:30975/TCP   2m9skubernetes                    ClusterIP      10.100.0.1       <none>                                                                    443/TCP        4h1mproducts-api                  ClusterIP      10.100.25.42     <none>                                                                    9090/TCP       73spublic-api                    ClusterIP      10.100.147.110   <none>                                                                    8080/TCP       108s

When the DNS successfully refreshes the hostname association for the frontend service, the HashiCups front page loads, containing a graphical carousel of different types of HashiCups beverages.

A picture of the HashiCups Portal, with a Packer Spiced Latte

Database can be written to by the frontend

In the portal, select a beverage and click on the Buy button. Enter in fake credit card information and press Submit Payment.

When you receive a status of "Payment processed successfully, card details returned for demo purposes, not for production", then the database in the primary cluster is successfully communicating with services across the Consul and Kubernetes clusters.

A picture of the payment portal for HashiCups, with a status message containing "Payment processed successfully, card details returned for demo purposes, not for production", indicating success.

Cleanup

Use terraform to remove the infrastructure from your account, now that this tutorial is complete. With AWS, this takes approximately 20 minutes.

$ cd .. && terraform destroy -auto-approve

Every effort has been made to make the cleanup of infrastructure resources in Terraform seamless. However, there are known issues with Terraform and the AWS provider, in which EKS creates resources for itself, not managed by Terraform, causing issues with removing all resources. A shim script in this repository mitigates this issue during terraform destroy. Due to timing of the Amazon API when removing resources, you may encounter issues where the infrastructure does not completely finish its removal. Inputting terraform destroy again should resolve the issue. But if not, you will need to remove the resources manually, from your AWS account.

Next steps

In this tutorial, you deployed a micro-services application, HashiCups, across multiple Kubernetes and Consul Enterprise clusters. You deployed a frontend service connected to a database deployed to another Kubernetes cluster, inside of another network. You confirmed that the client partition was registered to the server Consul Enterprise cluster.

To reiterate, Consul admin partitions give organizations the option to define and administer boundaries for services using Consul. This can assist organizations managing services across teams and business units. Teams can now benefit from managing and customizing their own Consul environment, without any impact to other Consul environments.

To learn more about admin partitions:

Read the Admin Partitions API documentation page
Take the Multi-Tenancy with Administrative Partitions tutorial
Read the Consul Enterprise Admin Partitions documentation page
Read the Achieving Multi-Tenancy with Consul Administrative Partitions blog post

Collection Overview

Implement multi-tenancy

Namespaces