Skip to content

Conversation

@upodroid
Copy link
Member

This PR introduces a few changes:

  1. Adds a new Terraform layer called root that handles entra id and privileged resources such as subscriptions, management groups
  2. Allows Atlantis to access Azure via WI
  3. All prow CI runs in the same sub where our build cluster runs. This PR creates the scaffolding to move CI to its own subscription and allows it to run directly from our build cluster.

@k8s-ci-robot k8s-ci-robot added area/infra Infrastructure management, infrastructure design, code in infra/ area/infra/azure Issues or PRs related to Kubernetes Azure infrastructure area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. labels Jan 20, 2026
@k8s-ci-robot k8s-ci-robot requested review from aojea and jbpratt January 20, 2026 21:04
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Jan 20, 2026
@k8s-infra-ci-robot

This comment has been minimized.

@k8s-infra-ci-robot

This comment has been minimized.

@k8s-infra-ci-robot
Copy link
Contributor

k8s-infra-ci-robot commented Jan 20, 2026

Argo CD Diff Preview

Summary:

Total: 2 files changed

Modified (2):
± aks-prow-build (+16)
± atlantis (+21)
aks-prow-build (kubernetes/apps/prow.yaml)
@@ Application modified: aks-prow-build (kubernetes/apps/prow.yaml) @@
       }
     }
 kind: ConfigMap
 metadata:
   name: google-adc
   namespace: test-pods
 ---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  annotations:
+    azure.workload.identity/client-id: 333bb18b-207b-4abd-9ed0-e7e3834378b1
+  name: azure
+  namespace: test-pods
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  annotations:
+    azure.workload.identity/client-id: f23f8fcc-855b-40fd-a41b-b329ccdb95a1
+  name: rg-cleanup
+  namespace: test-pods
+---
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
   annotations:
     api-approved.kubernetes.io: https://github.com/kubernetes-sigs/boskos/pull/105
   name: dynamicresourcelifecycles.boskos.k8s.io
 spec:
atlantis (kubernetes/apps/atlantis.yaml)
@@ Application modified: atlantis (kubernetes/apps/atlantis.yaml) @@
       - env:
         - name: AWS_ROLE_ARN
           value: arn:aws:iam::348685125169:role/atlantis
         - name: AWS_WEB_IDENTITY_TOKEN_FILE
           value: /var/run/secrets/aws-iam-token/serviceaccount/token
         - name: AWS_REGION
           value: us-east-2
+        - name: ARM_USE_AKS_WORKLOAD_IDENTITY
+          value: "true"
+        - name: ARM_SUBSCRIPTION_ID
+          value: 46678f10-4bbb-447e-98e8-d2829589f2d8
+        - name: AZURE_CLIENT_ID
+          value: 6fe87cee-6470-45d8-accc-57687193e504
+        - name: AZURE_FEDERATED_TOKEN_FILE
+          value: /var/run/secrets/azure-token/serviceaccount/token
+        - name: AZURE_TENANT_ID
+          value: d1aa7522-0959-442e-80ee-8c4f7fb4c184
         - name: ATLANTIS_CONFIG
           value: /config/atlantis.yaml
         - name: ATLANTIS_GH_TOKEN
           valueFrom:
             secretKeyRef:
               key: token
               name: atlantis-vcs
@@ skipped 32 lines (41 -> 72) @@
             memory: 1Gi
         volumeMounts:
         - mountPath: /config
           name: config
         - mountPath: /var/run/secrets/aws-iam-token/serviceaccount
           name: aws-iam-token
           readOnly: true
+        - mountPath: /var/run/secrets/azure-token/serviceaccount
+          name: azure-token
+          readOnly: true
         - mountPath: /atlantis
           name: atlantis-data
       securityContext:
         fsGroup: 1000
       serviceAccountName: atlantis
       volumes:
       - configMap:
           name: atlantis-config-4mc949mdm2
         name: config
       - name: aws-iam-token
         projected:
           defaultMode: 420
           sources:
           - serviceAccountToken:
               audience: sts.amazonaws.com
+              expirationSeconds: 86400
+              path: token
+      - name: azure-token
+        projected:
+          defaultMode: 420
+          sources:
+          - serviceAccountToken:
+              audience: api://AzureADTokenExchange
               expirationSeconds: 86400
               path: token
   updateStrategy:
     rollingUpdate:
       partition: 0
     type: RollingUpdate
   volumeClaimTemplates:

Stats:
[Applications: 74], [Full Run: 1m34s], [Rendering: 1m29s], [Cluster: 0s], [Argo CD: 1s]

@upodroid
Copy link
Member Author

atlantis plan

@k8s-infra-ci-robot

This comment has been minimized.

@upodroid
Copy link
Member Author

atlantis plan

@k8s-infra-ci-robot

This comment has been minimized.

@upodroid
Copy link
Member Author

atlantis plan

@upodroid upodroid removed the request for review from aojea January 20, 2026 22:15
@k8s-infra-ci-robot

This comment has been minimized.

@upodroid
Copy link
Member Author

atlantis plan

@k8s-infra-ci-robot

This comment has been minimized.

@upodroid
Copy link
Member Author

atlantis plan

@k8s-infra-ci-robot
Copy link
Contributor

Ran Plan for dir: infra/azure/terraform/root workspace: default

Plan Error

Show Output
running 'sh -c' '/usr/local/bin/terraform plan -input=false -refresh -out "/atlantis/repos/kubernetes/k8s.io/8974/default/infra/azure/terraform/root/default.tfplan"' in '/atlantis/repos/kubernetes/k8s.io/8974/default/infra/azure/terraform/root': exit status 1
Acquiring state lock. This may take a few moments...
module.role_assignments.data.modtm_module_source.telemetry[0]: Reading...
module.role_assignments.random_uuid.telemetry[0]: Refreshing state... [id=016e026a-dbc2-ff65-38c6-8554c6161445]
module.role_assignments.data.modtm_module_source.telemetry[0]: Read complete after 0s
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["owner"]: Reading...
module.role_assignments.data.azurerm_client_config.telemetry[0]: Reading...
module.role_assignments.data.azurerm_client_config.current: Reading...
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["contributor"]: Reading...
module.role_assignments.data.azurerm_management_group.management_groups_by_id_or_display_name["root"]: Reading...
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["monitoring-reader"]: Reading...
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["reader"]: Reading...
module.role_assignments.data.azurerm_client_config.current: Read complete after 0s [id=Y2xpZW50Q29uZmlncy9jbGllbnRJZD02ZmU4N2NlZS02NDcwLTQ1ZDgtYWNjYy01NzY4NzE5M2U1MDQ7b2JqZWN0SWQ9NDE4MDk0MTYtZTI4MC00OGUxLTgzNTQtODQ4M2FmMTkwZmJjO3N1YnNjcmlwdGlvbklkPTQ2Njc4ZjEwLTRiYmItNDQ3ZS05OGU4LWQyODI5NTg5ZjJkODt0ZW5hbnRJZD1kMWFhNzUyMi0wOTU5LTQ0MmUtODBlZS04YzRmN2ZiNGMxODQ=]
module.role_assignments.data.azurerm_client_config.telemetry[0]: Read complete after 0s [id=Y2xpZW50Q29uZmlncy9jbGllbnRJZD02ZmU4N2NlZS02NDcwLTQ1ZDgtYWNjYy01NzY4NzE5M2U1MDQ7b2JqZWN0SWQ9NDE4MDk0MTYtZTI4MC00OGUxLTgzNTQtODQ4M2FmMTkwZmJjO3N1YnNjcmlwdGlvbklkPTQ2Njc4ZjEwLTRiYmItNDQ3ZS05OGU4LWQyODI5NTg5ZjJkODt0ZW5hbnRJZD1kMWFhNzUyMi0wOTU5LTQ0MmUtODBlZS04YzRmN2ZiNGMxODQ=]
module.role_assignments.modtm_telemetry.telemetry[0]: Refreshing state... [id=af03c3da-3051-4ba2-b39b-f9d89929dbf6]
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["reader"]: Read complete after 0s [id=/providers/Microsoft.Authorization/roleDefinitions/acdd72a7-3385-48ef-bd42-f606fba81ae7]
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["contributor"]: Read complete after 0s [id=/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c]
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["owner"]: Read complete after 0s [id=/providers/Microsoft.Authorization/roleDefinitions/8e3af657-a8ff-443c-a75c-2fe8c4bcb635]
module.role_assignments.data.azurerm_role_definition.role_definitions_by_name["monitoring-reader"]: Read complete after 0s [id=/providers/Microsoft.Authorization/roleDefinitions/43d0d8ad-25c7-4714-9337-8ba259a9fe05]
module.role_assignments.data.azurerm_management_group.management_groups_by_id_or_display_name["root"]: Read complete after 0s [id=/providers/Microsoft.Management/managementGroups/d1aa7522-0959-442e-80ee-8c4f7fb4c184]

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: unable to build authorizer: no Authorizer could be configured, please check your configuration
│ 
│   with provider["registry.terraform.io/hashicorp/azuread"],
│   on providers.tf line 59, in provider "azuread":
│   59: provider "azuread" {
│ 
╵
Releasing state lock. This may take a few moments...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/infra/azure Issues or PRs related to Kubernetes Azure infrastructure area/infra Infrastructure management, infrastructure design, code in infra/ area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants