-
Notifications
You must be signed in to change notification settings - Fork 1.5k
CORS-4072: [Draft] Dual stack support for AWS #9930
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
@tthvo: This pull request references CORS-4072 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target either version "4.21." or "openshift-4.21.", but it targets "openshift-4.20" instead. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/cc @mtulio Just rough hacks but in case you are interested :D |
|
@tthvo: This pull request references CORS-4072 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target either version "4.21." or "openshift-4.21.", but it targets "openshift-4.20" instead. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
0e37a46 to
537f4d0
Compare
|
/test e2e-aws-default-config e2e-aws-ovn-shared-vpc-custom-security-groups |
|
/retitle CORS-4072: [Draft] Dual stack support for AWS This PR is for experimenting and collecting info about what changes are needed. I will separate the commits into smaller PRs :D PTAL 🙏 All reviews and nitpicks are appreciated! |
d5cef48 to
dac6b81
Compare
|
The rebase is to stay on top of |
|
I rebuilt another release image: quay.io/thvo/origin-release:v4.21.0-preview-1. This includes the changes for openshift/cluster-network-operator#2804 instead of my own hack tthvo/cluster-network-operator@617e05f. If you'd like to use the new custom release image, you need to set the techpreview feature set: featureSet: TechPreviewNoUpgrade |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@tthvo: This pull request references CORS-4072 which is a valid jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@tthvo: This pull request references CORS-4072 which is a valid jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
The PR is rebased on top of main with latest capa IPv6 PR changes; and adjusted to match #10207 implementation for the install-config field. I also rebuilt a new custom release image based on Both dual-stack Ipv4 primary and dual-stack IPv6 primary install (using install-config in the PR description) should proceed to the end successfully (and "seamlessly") 😄 |
patrickdillon
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good on first pass
Based on install-config input, update AWSPlatformStatus's IPFamily field within the Infrastructure manifest. Update unit tests to reflect this new field.
…-network-server The commit ensures all service networks are considered (i.e. that is all IP families) when generating the certificate kube-apiserver-service-network-server.
The installconfig in the cluster-config ConfigMap needs to have the Ipv6 CIDR of the VPC in the case of full IPI.
This applies to dualstack installation only. IPv4-primary: IPv4 Target Group IPv6-primary: Ipv6 Target Group
FIXME: we should use the VPC CIDR as the source CIDRs. But the IPv6 cidr is not yet knowned at install time. We should edit the awscluster after infraReady to add the VPC IPv6 CIDR as source instead.
FIXME: CCM and in-cluster MAPI/CAPI needs to handle this
|
@tthvo: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
I have started on splitting the changes in this PR into smaller individual PRs for easy review. But this PR is still kept open for quick testing. |
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Important
A rough draft of installer changes required to support dual-stack environment on AWS.
This PR is only for previewing the changes and experimenting with upstream CAPA PR. I will close this and open another PR with finalized sets of changes.
This PR also includes commits (message starting with
hack:) to "imitate" CCM, MAPI, and Cluster Ingress Operator to create necessary resources for cluster ingress (i.e. NLB, Route53 records, Security Groups, etc) and enable Ipv6 primary (if applicable). These commits are to be removed, assuming dual-stack is supported in operators later on.This depends on upstream CAPA PR: kubernetes-sigs/cluster-api-provider-aws#5603
How to install
Below is the details of how to reproduce the installation.
Custom release image
Custom release image:
quay.io/thvo/origin-release:v4.22.0-preview-dsThis includes the following operator changes:
For the
cluster-network-operator, we have the open PR here with feature gate checking: openshift/cluster-network-operator/pull/2804Install Config
Use the below
install-configsnippet to configure networking and AWS platform.Note:
machineNetworkdoes not contain IPv6 CIDR as it is unknown at install time (i.e. will be patched later when infra is ready). The cluster network and service network contain ULA IPv6 CIDR.IPv4 Primary:
IPv6 Primary:
Important notes: [IPv6-primary only] The ingress operator will be stuck as health check on targets are failing because the k8s Service for ingress routers only have IPv6 cluster IP. The hacks only configures the ingress LB target group as IPv4, thus the connection cannot switch to IPv6 when travelling internally.You must edit the that serviceopenshift-ingress/router-nodeport-defaultto set itsipFamilyPolicytoPreferDualStack. For example:$ kubectl -n openshift-ingress patch svc router-nodeport-default \ -p '{"spec":{"ipFamilyPolicy":"PreferDualStack"}}'Updated: There is a new commit to "hack" enable IPv6 primary on ec2 instances for cluster nodes. So, above step is no longer needed. The Target Group for dual-stack IPv6 primary is now also IPv6.
Installer binary
The installer binary can be built normally from these commits (i.e. capa is vendored from my fork). So, just:
/hold
/label platform/aws