Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EKS] [request]: Add ability to set oidc options #166

Closed
jaxxstorm opened this issue Feb 15, 2019 · 62 comments
Closed

[EKS] [request]: Add ability to set oidc options #166

jaxxstorm opened this issue Feb 15, 2019 · 62 comments
Labels
EKS Amazon Elastic Kubernetes Service Proposed Community submitted issue

Comments

@jaxxstorm
Copy link

Tell us about your request
Using IAM for authentication is not always the most desirable solution. I'd like to be able to specify server side configuration, specifically for oidc authentication

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
When interacting with EKS clusters from non Amazon resources, having to create and specify access keys and secret keys can be cumbersome. We do this a lot at $work and because we can't specify an oidc configuration for the API server, we end up having to build our own clusters.
Being able to set the following API flags on the api server:

  • oidc-issuer-url
  • oidc-client-id
  • oidc-ca-file
  • oidc-username-claim
  • oidc-groups-claim

Would make this possible.

Are you currently working around this issue?
We are building our clusters using kubeadm :(

Additional context
This could be argued that it should be made possible to specify any API server flags, but I'd like to keep the scope smaller now

@jaxxstorm jaxxstorm added the Proposed Community submitted issue label Feb 15, 2019
@abby-fuller abby-fuller added the EKS Amazon Elastic Kubernetes Service label Feb 15, 2019
@christopherhein
Copy link

Thanks for submitting this. This is not currently on our roadmap.

Can you help me understand more about you use case, would https://aws.amazon.com/blogs/opensource/integrating-ldap-ad-users-kubernetes-rbac-aws-iam-authenticator-project/ help to solve this?

It would give you the ability to use your AWS SSO to back your IAM roles which you could configure to use any LDAP/AD/SAML authentication mechanisms.

@christopherhein christopherhein added Not planned and removed Proposed Community submitted issue labels Feb 15, 2019
@jaxxstorm
Copy link
Author

This is possible yes, but when using tooling like Azure Active Directory, it add an additional layer of configuration.

Currently, as a user using oidc, I can:

  • Assign the user to a group in Azure AD
  • Map that group to a Kubernetes RBAC role

However, under the model you're proposing, I'd have to:

  • Assign a user to a group in Azure AD
  • Map that user to an IAM role, presumably with role assumption
  • Map the IAM role to a Kubernetes RBAC role

This additional layer adds complexity for debugging purposes. For example - if a user has a permission issue, I have to track the issue down by checking their AD group, then checking their IAM role, then RBAC. It seems unnecessary to me to not just add the API server flag.

@lstoll
Copy link

lstoll commented Feb 19, 2019

FWIW we ended up deploying an auth proxy to our clusters that authenticates users via OIDC, and then impersonates them to the API server. That allows us to manage access via our existing systems for users that may not have AWS access otherwise. The code is vaguely based on https://github.com/ericchiang/kube-oidc#kube-oidc-proxy

@christopherhein
Copy link

FWIW we ended up deploying an auth proxy to our clusters that authenticates users via OIDC, and then impersonates them to the API server. That allows us to manage access via our existing systems for users that may not have AWS access otherwise. The code is vaguely based on https://github.com/ericchiang/kube-oidc#kube-oidc-proxy

Sounds neat, is the code open source anywhere?

@jaxxstorm
Copy link
Author

@lstoll that sounds ideal - I would love some help/pointers. I really do think this should be reconsidered by AWS, so upvotes would be awesome if people want this feature

@sermilrod
Copy link

sermilrod commented Feb 22, 2019

I am also interested in this feature. The ability to configure identity providers is something I would like EKS to support. I need this because we want to map specific users to rbac rules so we have full and granular control over who has what access to kubernetes clusters without having to create an IAM user nor assume a role for it. This is particularly important in multi-tenant clusters.

@universam1
Copy link

FWIW we ended up deploying an auth proxy to our clusters that authenticates users via OIDC, and then impersonates them to the API server. That allows us to manage access via our existing systems for users that may not have AWS access otherwise.

We do the similar as well, but the OIDC integration in K8s is required still in order to map the JWT token into RBAC roles. For instance Kubernetes Dashboard needs that mapping to gain cluster level access.
@christopherhein this missing support is a surprise no-go for us in the effort moving to EKS!

@christopherhein
Copy link

Thank you everyone for the feedback, really appreciate you talking about your use cases. Changing the label to under consideration 👍

@jaxxstorm
Copy link
Author

@christopherhein would it be easier to get this considered if we filed an issue which allows us to specify API server configuration in general, rather than specifically for oidc flags?

@christopherhein
Copy link

@christopherhein would it be easier to get this considered if we filed an issue which allows us to specify API server configuration in general, rather than specifically for oidc flags?

These more specific issues really help us to prioritize what is most important. The larger "enable all API flags configurable" are much harder given there is so many controls that could introduce instability if improperly set if everything was customizable. Appreciate you putting this in and sharing the use cases.

@universam1
Copy link

To second that point of only requesting this very feature for now - while it is rather simple to configure the oidc configuration it is completely indispensable regarding alternatives!
IAM auth can be considered an option for tooling access but application level that is no alternative.
Is there an estimation possible @christopherhein ? Would be helpful for planning

@christopherhein
Copy link

Estimation for this feature of adding OIDC flag capability @universam1 ? or are you talking about the Pod->IAM configuration #23 ?

@pammi22
Copy link

pammi22 commented Mar 6, 2019

@christopherhein is there any Estimation for this feature of adding OIDC flag capability?

@christopherhein
Copy link

@pammi22 not as of now, we removed the not-planned label in favor of the under consideration while we consider how we would enable this customization with the control plane.

@whereisaaron
Copy link

The lack of OIDC support for EKS is our single biggest issue for adoption. With stock k8s and dex we have a very clean, two-factor, federated single sign-on for clusters. These flags requested here would enable EKS clusters to have the same great user experience.

If EKS had built-in AWS Cognito support it is possible we could replace dex with Cognito for federating OIDC IDPs. But that support isn’t there currently AFAIK, and I can’t find any existing helper app that might bridge this gap in EKS. Anyone know of a project like this?

@aleks-mariusz
Copy link

aleks-mariusz commented Mar 29, 2019

I would like to echo all the sentiments..

We use AD for corporate identity management, and to access AWS console we let the infra/admin group (only) use Okta which loads the console using a dedicated IAM role that grants full access. It is unclear to me how setting up EKS with this assumed IAM role would affect having cluster-admin privs, but wanting to give end-users (sandbox'd) permissions to the kubernetes API server is very messy using EKS.

For contrast, what we currently have on-prem is AD -> federation via Keycloak -> which integrates with the k8s dashboard via the proxy -> k8s dashboard.. and this allows end-users to log in with their AD creds cleanly, which works by mapping via JWT's provided by keycloak (which contain their AD group membership), that membership can then be mapped directly via RBAC rolebindings, to what roles they have (which control what access they have).

It is such a clean and tight integration that it makes us want to consider skipping using EKS altogether and having a cloud-based kubernetes cluster by just managing the control plane ourselves. I have yet to see what using EKS gets you over just rolling your own control plane if we don't have access to provide consistency (auth-wise) with what we have on-prem. :-( Allowing us to add those 4 extra oidc related flags [ oidc-client-id / oidc-issuer-url / oidc-username-claim / oidc-groups-claim ] would have us giving EKS more serious consideration for use.

@Spazzy757
Copy link

Has there been an update to this issue? We are currently doing a POC for quite a large company to prove that they can move there on-prem clusters to EKS, however, they insist that they would need AD integration. Adding those flags could open up a much larger consumer base for EKS as it would allow migration from systems like AKS where the system is integrated into Azure Active Directory

@philwhln
Copy link

We had to cancel our PoC of EKS because of this limitation.

@ccojocar
Copy link

Any ETA for this feature? Thanks

@phanama
Copy link

phanama commented May 24, 2019

Any updates?
This limitation hesitates us from adopting EKS

@tabern tabern added Proposed Community submitted issue and removed Under consideration labels Jun 4, 2019
@tabern tabern added this to We're Working On It in containers-roadmap Jun 5, 2019
@jlforester
Copy link

jlforester commented Apr 6, 2020

I've been looking at kubelogin which needs these two flags added to the apiserver

--oidc-issuer-url=ISSUER_URL
--oidc-client-id=YOUR_CLIENT_ID

@salamovz
Copy link

salamovz commented Apr 7, 2020

Yes it perfectly work on EKS.
I changed gen-certs.sh to be:
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout server-key.pem -out server.pem -subj "/CN=kube-oidc-proxy.kube-oidc-proxy" -extensions SAN -config <(printf "[ req_distinguished_name ]\n[req]\ndistinguished_name = req_distinguished_name\n[SAN]\nsubjectAltName='DNS.1:kube-oidc-proxy.kube-oidc-proxy'")
But you can use any other tool to generate certificate.
In args inside of kubectl -n kube-oidc-proxy edit deploy oidc-proxy-dashboard
I added: - --scopes=groups
And you DONT need to uncomment --apiserver-host vi +125 dashboard/kubernetes-dashboard.yaml
And when creating secrets please ensure you added '-n' to echo while creating base64 values like:
echo -n "email" | base64
E.g kube-oidc-proxy/secrets.yaml

data:
  oidc.ca-pem: YW55RHVtbXlWYWx1ZUlmWW91VXNlVmFsaWRDZXJ0T25EZXg=
  oidc.issuer-url: eW91ckRFWG5hbWUudXJsLndpdGgudmFsaWQuY2VydA==
  oidc.username-claim: ZW1haWw=
  oidc.client-id: a3ViZXJuZXRlcw==
  oidc.client-secret: WlhoaGJYQnNaUzFoY0hBdGMyVmpjbVYw

@enj
Copy link

enj commented Apr 15, 2020

I have opened kubernetes/enhancements#1689 as a provisional KEP to address this issue in a generic manner.

@abdellaui
Copy link

Really necessary feature for production environments for avoiding unsupported work arounds😔

@pierluigilenoci
Copy link

There is an open issue since one year for AWS SSO support on boto3.
boto/boto3#2091

@halcyondude
Copy link

bump. we're grappling with this presently. Any update from the EKS team?

@Tokynet
Copy link

Tokynet commented Aug 24, 2020

...Bueller?
Any update or acknowledgment from AWS team would be appreciated.

@mhausenblas
Copy link
Member

Any update or acknowledgment from AWS team would be appreciated.

I hear you. Lemme check back with the team and see if I can confirm something.

(PS: the 80s want their lines back ;)

@mikestef9 mikestef9 moved this from We're Working On It to Coming Soon in containers-roadmap Aug 24, 2020
@mikestef9
Copy link
Contributor

Hey all, just moved status to "coming soon". One important note here, is that EKS will only support issuer URLs with a publicly signed CA certificate, and oidc-ca-file will not be a supported flag.

@alex-berger
Copy link

@mikestef9 Why is setting oidc-ca-file not supported, is there a technical reason or is it just not in scope for the first MVP? And which trust anchors (root CAs) will be used by the control plane to verify the oidc issuer's server certificate?

@degonzalezo
Copy link

Any news? Would like to start using this configuration as the current state via AWS SSO leads to undesired drifts when managing multiple cluster environments at scale.

@mercantiandrea
Copy link

any news?

@cdobbyn
Copy link

cdobbyn commented Dec 7, 2020

It's not a high priority for them. They seemed pretty dismissive of it and suggested kube-oidc-proxy. It's pretty debilitating for administrators to have to link this to IAM. This becomes ultra complicated when you're managing multiple tenants on multiple clusters whom may need access to subsets of namespaces, across a number of aws accounts.

It sounds like the kind of priority that will keep getting shifted down to the bottom of the priority list.

@Morriz
Copy link

Morriz commented Feb 5, 2021

I don't think AWS is interested in supporting other cloud users...out of the big cloud players, they seem to favor lock in the most...I am assuming they allow AWS Cognito via oidc?

@pierluigilenoci
Copy link

@Morriz Cognito via OIDC is already working but it is annoying because every time you have to go through tokens with very short expiration and through the CLI. And you need to change the cluster config deeply to integrate with AAD groups. :(

@Morriz
Copy link

Morriz commented Feb 8, 2021

@pierluigilenoci that is what I meant: they support their own (Cognito), but make it hard to use others (AAD).

@mikestef9
Copy link
Contributor

mikestef9 commented Feb 13, 2021

Hey everyone, happy to announce EKS now supports OIDC compatible identity providers as an additional user authentication option! The launch blog below walks through using Cognito as an example, but you can use any compatible OIDC provider as long as the issuer URL is publicly available with a commercially signed CA.

@Freyert
Copy link

Freyert commented Jun 24, 2022

Does anyone have tips on troubleshooting this feature? The configuration is active, but when we submit a kubectl with a --token the audit log shows that the server did not populate the user field of the request.

The release blog shows the request should have a user field like so:

    "user": {
        "username": "test@example.com",
        "groups": [
            "gid:secret-reader",
            "system:authenticated"
        ]
    },

But our request always turn up like:

    "user": {
    },

And that's as far as I can get. I don't know of any other logs to look at to troubleshoot this issue.


It look like the apiserver handles these requests itself. Found these:

E0624 17:03:16.231600 9 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, oidc: verify token: oidc: id token signed with unsupported algorithm, expected [\"RS256\"] got \"ES256\", unknown]"

🤞 that's the issue.


Update: it was and it works now. If you are also debugging this feature look at the apiserver log streams in cloudwatch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
EKS Amazon Elastic Kubernetes Service Proposed Community submitted issue
Projects
containers-roadmap
  
Just Shipped
Development

No branches or pull requests