[FCP] [AAC-AKS] customer facing readme md - Part 2 (#336)

improve README file Co-authored-by: Chad Kittel <chad.kittel@gmail.com>
mspnp · Jun 29, 2020 · e328804 · e328804
1 parent 4a58721
commit e328804
Show file tree

Hide file tree

Showing 11 changed files with 768 additions and 54 deletions.
diff --git a/README.md b/README.md
diff --git a/aad/aad.azcli b/aad/aad.azcli
@@ -0,0 +1,38 @@
+# The Contoso Bicycle Azure AD Admin team has a new organizational challenge
+# which is regulating access to the new Secure AKS cluster (Application ID: a0008 under the BU001)
+# resources from Azure AD.
+# The goal is being able to configure Kubernetes role-based access control (RBAC)
+# based on a user's identity or directory group membership.
+# Thefore, the team has made the decision of implementing the Azure AD integration with
+# AKS-managed Azure AD.
+
+# The user admin logins into the tenant
+az login --allow-no-subscriptions -t $K8S_RBAC_AAD_PROFILE_TENANTID && \
+K8S_RBAC_AAD_PROFILE_TENANT_DOMAIN_NAME=$(az ad signed-in-user show --query 'userPrincipalName' | cut -d '@' -f 2 | sed 's/\"//')
+
+# They create first the group that is going to map the Kubernetes Cluster Role Admin.
+# Once the cluster gets deployed the new group will get the proper Cluster
+# Role bindings
+K8S_RBAC_AAD_ADMIN_GROUP_OBJECTID=$(az ad group create --display-name add-to-bu0001a000800-cluster-admin --mail-nickname add-to-bu0001a000800-cluster-admin --query objectId -o tsv)
+
+# Later the App team's admin member requested a Cluster Admin User. Therefore,
+# the Azure AD Admin team procceds to the creation of a new user from Azure AD.
+AKS_ADMIN_OBJECTID=$(az ad user create --display-name=bu0001a0008-admin --user-principal-name bu0001a0008-admin@${K8S_RBAC_AAD_PROFILE_TENANT_DOMAIN_NAME} --force-change-password-next-login --password bu0001a0008Admin --query objectId -o tsv)
+
+# Then the recently created user is added to the Kubernetes Cluster Admin group
+# from Azure AD
+az ad group member add --group add-to-bu0001a000800-cluster-admin --member-id $AKS_ADMIN_OBJECTID
+
+# Later on the Azure AD Admin team will review the baseline settings of the Secure AKS cluster (Application ID: a0008 under the BU001)
+# and edit the file:
+#   - `aks/security-baseline/cluster-baseline-settings/user-facing-cluster-role-aad-group.yaml`
+# to replace the placeholder:
+#   - `<replace-with-an-aad-group-object-id-for-k8s-admin-clusterrole>`
+# using the object id for the recently added Azure AD group.
+
+# Lastly, the Azure AD respond to the oginal App team's user creation requet by
+# opening a PR againts the Secure AKS Cluster (Application ID: a0008 under the BU001) IaaC git repository.
+
+# At this moment the Azure AD Admin team hands off this process to the networking team
+# so they could deliver the required networking assets to lay down a Secure
+# AKS cluster (Application ID: a0008 under the BU001)
diff --git a/cluster-deploy.azcli b/cluster-deploy.azcli
@@ -13,24 +13,10 @@ az group create --name rg-bu0001a0008 --location eastus2
 # [This takes about 15 minutes.]
 az deployment group create --resource-group rg-bu0001a0008 --template-file cluster-stamp.json --parameters "@azuredeploy.parameters.prod.json"
 
-# Get AKS Kubeconfig Credntials
-az aks get-credentials -n [cluster-name] -g rg-bu0001a0008 --admin
-
-
-# The Azure Active Directory Admin team wants to grant access to the new created
-# cluster. Initially for AAD Bicycle Contoso Tenant Administrators. Therefore a new
-# group is created in the AAD that represents the Kubernetes Cluster Admin Role.
-# Later on the AAD Admin team will review the baseline settings of the cluster and
-# edit the `aks/security-baseline/cluster-baseline-settings/user-facing-cluster-role-aad-group.yaml`
-# to replace the placeholder `replace-with-an-aad-group-object-id-for-k8s-admin-clusterrole`
-# with the object id for the recently added AAD group.
-
-# Deploy flux
-kubectl create namespace cluster-baseline-settings
-kubectl apply -f https://raw.githubusercontent.com/mspnp/reference-architectures/master/aks/secure-baseline/cluster-baseline-settings/flux.yaml
-kubectl wait --namespace cluster-baseline-settings \
-  --for=condition=ready pod \
-  --selector=app.kubernetes.io/name=flux \
-  --timeout=90s
-
-
+# Finally the app team wants to import a wildcard certificate *.aks-ingress.contoso.com to AzureKeyVault
+# A while later this certificate is going to be the one served by a Traefik Ingress Controller wich is
+# deployed downstream
+KEYVAULT_NAME=$(az deployment group show --resource-group rg-bu0001a0008 -n cluster-stamp --query properties.outputs.keyVaultName.value -o tsv) && \
+az keyvault set-policy --certificate-permissions import -n $KEYVAULT_NAME --upn $(az account show --query user.name -o tsv) && \
+cat traefik-ingress-internal-aks-ingress-contoso-com-tls.crt traefik-ingress-internal-aks-ingress-contoso-com-tls.key > traefik-ingress-internal-aks-ingress-contoso-com-tls.pem && \
+az keyvault certificate import --vault-name $KEYVAULT_NAME -f traefik-ingress-internal-aks-ingress-contoso-com-tls.pem -n traefik-ingress-internal-aks-ingress-contoso-com-tls
diff --git a/0-networking-stamp.sh → deploy/0-networking-stamp.sh b/0-networking-stamp.sh → deploy/0-networking-stamp.sh
diff --git a/1-cluster-stamp.sh → deploy/1-cluster-stamp.sh b/1-cluster-stamp.sh → deploy/1-cluster-stamp.sh
diff --git a/deleteResourceGroups.sh → deploy/deleteResourceGroups.sh b/deleteResourceGroups.sh → deploy/deleteResourceGroups.sh
diff --git a/deploy/readme.md b/deploy/readme.md
@@ -0,0 +1,21 @@
+You can choose to deploy the Secure AKS cluster baseline by executing the following script files.
+
+> Tip: we recommend to deploy this Reference Implementation using the README.md
+> steps, but please feel free to use this deployment path if you find
+> this more convinient
+
+### Deploy
+
+> Important: edit these script files to complete the required values before procedding
+
+```bash
+# [This takes thirty minutes to run.]
+./0-networking-stamp.sh && \
+./1-cluster-stamp.sh`
+
+### Clean up
+
+```bash
+# [This takes twenty minutes to run.]
+./deleteResourceGroups.sh
+```
diff --git a/networking-readme.md b/networking-readme.md
@@ -29,7 +29,7 @@ subnets, more [AKS Nodepools subnets], [Private endpoints], and more.
 
 | Subnet                                                 | Upgrade Node | Nodes/VMs/Instance | % Xmas scale out | +Nodes/VMs | Max Ips/Pods per VM/Node | [% Max Surge] | [% Max Unavailable] | +Ips/Pods per VM/Node | Tot. Ips/Pods per VM/Node | [Azure Subnet not assignable Ips factor] | [Private Endpoints] | [Minimum Subnet size] | Scaled Subnet size | [Subnet Mask bits] | Cidr           | Host        | Broadcast     |
 |--------------------------------------------------------|--------------|--------------------|------------------|------------|--------------------------|---------------|---------------------|-----------------------|---------------------------|------------------------------------------|---------------------|-----------------------|--------------------|--------------------|----------------|-------------|---------------|
-| AKS System and User Nodepool Subnet                    | 1            | 3                  | 400              | 12         | [30]                     | 100           | 0                   | 30                    | 60                        | 5                                        | 0                   | 249                   | 981                | 22                 | 10.240.0.0/22  | 10.240.0.0  | 10.240.3.255  |
+| AKS System and User Nodepool Subnet                    | 1            | 5                  | 200              | 10         | [30]                     | 100           | 0                   | 30                    | 60                        | 5                                        | 2                   | 373                   | 983                | 22                 | 10.240.0.0/22  | 10.240.0.0  | 10.240.3.255  |
 | AKS Internal Load Balancer Services Subnet             | 0            | 0                  | 0                | 0          | 5                        | 100           | 100                 | 0                     | 5                         | 5                                        | 0                   | 10                    | 10                 | 28                 | 10.240.4.0/28  | 10.240.4.0  | 10.240.4.15   |
 | Azure Application Gateway Subnet                       | 0            | [11]               | 0                | 0          | 0                        | 100           | 100                 | 0                     | 0                         | 5                                        | 0                   | 16                    | 16                 | 28                 | 10.240.4.0/28  | 10.240.4.16 | 10.240.4.31   |
 | Gateway Subnet (GatewaySubnet)                         | 0            | [27<sup>1</sup>]   | 0                | 0          | 0                        | 100           | 100                 | 0                     | 0                         | 5                                        | 0                   | 32                    | 32                 | 27                 | 10.200.0.64/27 | 10.200.0.64 | 10.200.0.95   |
@@ -40,6 +40,7 @@ subnets, more [AKS Nodepools subnets], [Private endpoints], and more.
 
 1. [AKS System Nodepool] and [AKS User Nodepool] subnet:  Multi-tenant or other advanced workloads may have nodepool isolation requirements that might demand more (and likely smaller) subnets.
 2. [AKS Internal Load Balancer subnet]: Multi-tenant, multiple SSL termination rules, single PPE supporting dev/QA/UAT, etc could lead to needing more ingress controllers, but for baseline, we should start with one.
+3. [Private Endpoints] Private Links are created for ACR and Azure KeyVault, so these Azure services can be accessed using Private Endpoints within the Spoke vNet, specifically allocating an private Ip address from the AKS System and User Nodepool Subnet.
 
 [27<sup>1</sup>]: https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpn-gateway-settings#gwsub
 [11]: https://docs.microsoft.com/en-us/azure/application-gateway/configuration-overview#size-of-the-subnet

diff --git a/readme.md b/readme.md
diff --git a/workload/aspnetapp.yaml b/workload/aspnetapp.yaml
@@ -0,0 +1,122 @@
+#  ------------------------------------------------------------
+#   Copyright (c) Microsoft Corporation.  All rights reserved.
+#   Licensed under the MIT License (MIT). See License.txt in the repo root #  for license information.
+#  ------------------------------------------------------------
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: aspnetapp-deployment
+  namespace: a0008
+  labels:
+    app.kubernetes.io/name: aspnetapp
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: aspnetapp
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: aspnetapp
+    spec:
+      automountServiceAccountToken: false
+      securityContext:
+        runAsUser: 10001
+        runAsGroup: 3000
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+          - labelSelector:
+              matchExpressions:
+              - key: app.kubernetes.io/name
+                operator: In
+                values:
+                - aspnetapp
+            topologyKey: "kubernetes.io/hostname"
+        podAffinity:
+          preferredDuringSchedulingIgnoredDuringExecution:
+          - weight: 1
+            podAffinityTerm:
+              labelSelector:
+                matchExpressions:
+                - key: app.kubernetes.io/name
+                  operator: In
+                  values:
+                  - traefik-ingress-ilb
+              topologyKey: "kubernetes.io/hostname"
+      containers:
+      - name: aspnetcore-webapp-sample
+        image: mcr.microsoft.com/dotnet/core/samples:aspnetapp
+        imagePullPolicy: Always
+        securityContext:
+          allowPrivilegeEscalation: false
+          readOnlyRootFilesystem: true
+          capabilities:
+            drop:
+              - all
+        env:
+        - name: ASPNETCORE_URLS
+          value: "http://*:8080"
+        - name: COMPlus_EnableDiagnostics
+          # https://github.com/dotnet/dotnet-docker/issues/940
+          value: "0"
+      nodeSelector:
+       agentpool: npuser01
+---
+apiVersion: policy/v1beta1
+kind: PodDisruptionBudget
+metadata:
+  name: aspnetapp-pdb
+spec:
+  minAvailable: 75%
+  selector:
+    matchLabels:
+      app: aspnetapp
+---
+kind: Service
+apiVersion: v1
+metadata:
+  name: aspnetapp-service
+  namespace: a0008
+spec:
+  selector:
+    app.kubernetes.io/name: aspnetapp
+  ports:
+  - name: http
+    port: 80
+    targetPort: 8080
+---
+apiVersion: networking.k8s.io/v1beta1
+kind: Ingress
+metadata:
+  name: aspnetapp-ingress
+  namespace: a0008
+  annotations:
+    # defines controller implementing this ingress resource: https://docs.microsoft.com/en-us/azure/dev-spaces/how-to/ingress-https-traefik
+    # ingress.class annotation is being deprecated in Kubernetes 1.18: https://kubernetes.io/docs/concepts/services-networking/ingress/#deprecated-annotation
+    # For backwards compatibility, when this annotation is set, precedence is given over the new field ingressClassName under spec.
+    kubernetes.io/ingress.class: traefik-internal
+    traefik.ingress.kubernetes.io/router.entrypoints: websecure
+    traefik.ingress.kubernetes.io/router.tls: "true"
+    traefik.ingress.kubernetes.io/router.tls.options: default
+    traefik.ingress.kubernetes.io/router.middlewares: app-gateway-snet@file, gzip-compress@file
+spec:
+  # ingressClassName: "traefik-internal"
+  tls:
+  - hosts:
+      - bu0001a0008-00.aks-ingress.contoso.com
+        # it is possible to opt for certificate management strategy with dedicated
+        # certificates for each TLS SNI route.
+        # In this Rereference Implementation for the sake of simplicity we use a
+        # wildcard default certificate added at Ingress Controller configuration level which is *.example.com
+        # secretName: <bu0001a0008-00-example-com-tls-secret>
+  rules:
+  - host: bu0001a0008-00.aks-ingress.contoso.com
+    http:
+      paths:
+      - path: /
+        backend:
+          serviceName: aspnetapp-service
+          servicePort: http
+---