API OIDC authentication mechanism #10905

ErykKul · 2024-10-03T15:33:54Z

What this PR does / why we need it:
It reimplements OIDC authentication mechanism (including the bearer tokens authentication)

Which issue(s) this PR closes:

Closes #10973

Special notes for your reviewer:
This is very similar conceptually as the oauth2-proxy, as in IQSS/dataverse-frontend#504, however, it turns out that the needed functionality is already supported in Payara: https://docs.payara.fish/enterprise/docs/Technical%20Documentation/Public%20API/OpenID%20Connect%20Support.html
This PR eliminates any need for a proxy, etc. It turned out that the implementation is very simple and elegant, i.m.h.o.

Suggestions on how to test this:

it is a good idea to test the entire flow and make sure the user you are testing with does not yet exist in the dev dataverse instance, but it does exist keycloack. You can delete your docker volumes with sudo rm -rf docker-dev-volumes, if that helps. Also, make sure that you have an entry for the Kaycloak in your hosts file (e.g., /etc/hosts on unix system), as described in the docker-compose file. It should look soemthing like this:

127.0.1.1       keycloak.mydomain.com

you can run the dev env with mvn -Pct clean package docker:run
go to http://localhost:8080/ and click 'Log In', click "OpenID Connect" and then click "Log In with OpenID Connect"
log in with admin/admin

you will get the redirect to create a new user (because of the first time log in)

choose a user name and agree to the conditions, click on "Create Account" and now you are logged in:

log out and log in with OpenID connect again, to make sure that works
test the API call with going in the browser to http://localhost:8080/api/v1/users/:me

test the session is working by going (still in the browser) to http://localhost:8080/api/v1/oidc/session

copy the "session" field and try curl (replace the session-id with the copied value)

curl -v --cookie "JSESSIONID=session-id" http://localhost:8080/api/v1/users/:me | jq .

test the API/Bearer token flow by running the Python script, e.g.:

cd doc/sphinx-guides/_static/api/bearer-token-example
./run.sh

this will start a Python script that will prompt a log in in a new browser window:

log in with admin/admin
Python will get the bearer token and call http://localhost:8080/api/v1/users/:me

Does this PR introduce a user interface change? If mockups are available, please link/include them here:
No

Is there a release notes update needed for this change?:
Yes

Additional documentation:
https://docs.payara.fish/enterprise/docs/Technical%20Documentation/Public%20API/OpenID%20Connect%20Support.html

Doc preview: https://dataverse-guide--10905.org.readthedocs.build/en/10905/installation/oidc.html

src/main/java/edu/harvard/iq/dataverse/api/OpenIDCallback.java

coveralls · 2024-10-03T15:46:12Z

coverage: 20.705% (-0.2%) from 20.869%
when pulling 0ec91dd on authn-arch-api-oidc
into a0cb73d on develop.

docker-compose-dev.yml

Thanks! Co-authored-by: Philip Durbin <[email protected]>

ErykKul · 2024-10-03T18:39:03Z

I forgot to mention this: the log-in is a one time thing. The normal flow is to go to http://localhost:8080/api/v1/callback/session and only if you are not authenticated (I implemented returning better error message now), you go to http://localhost:8080/oidc/login.

@pdurbin thanks for the docker run command, I will try testing it with that i.s.o. my own dev environment.

pdurbin

Some initial feedback. I haven't done any real testing yet.

docker-compose-dev.yml

src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java

src/main/java/edu/harvard/iq/dataverse/api/OpenIDAuthentication.java

src/main/java/edu/harvard/iq/dataverse/api/OpenIDCallback.java

…d in dv

ErykKul · 2024-10-08T13:48:21Z

I have tried this PKCE client implementation (added to doc/sphinx-guides/_static/frontend/PKCE-example/PKCE-example.html):

<!doctype html>
<html>

<body>
    <script src="http://unpkg.com/[email protected]/dist/keycloak-authz.js"></script>
    <script src="http://unpkg.com/[email protected]/dist/keycloak.js"></script>

    <script>
        const kc = new Keycloak({
            url: 'http://keycloak.mydomain.com:8090',
            realm: 'test',
            clientId: 'test'
        });
        kc.init({
            pkceMethod: 'S256',
            redirectUri: 'http://localhost:8080/api/v1/users/:me'
        });
        kc.login();
    </script>
</body>

</html>

At first it looked like it worked perfectly, but then I noticed I was redirected to a different page than intended... Then I checked the network log and it looks a lot like what auth2-proxy would do:

Then I tried http://localhost:8080/api/v1/users/:me in the browser (when not being logged in), and very similar thing happened; this really behaves like an oath2-proxy. I am not sure if I like it. At least it did not break PKCE clients, so to see.

Anyway, I am putting this on hold, until we have a better understanding what we want to do.

github-actions · 2024-10-09T18:06:31Z

doc/sphinx-guides/_static/api/bearer-token-example/run.sh

+#!/bin/bash
+
+python3 -m venv run_env
+source run_env/bin/activate


📝 [shellcheck] _{reported by reviewdog 🐶}
Not following: run_env/bin/activate: openBinaryFile: does not exist (No such file or directory) SC1091

ErykKul · 2024-10-13T14:13:37Z

I have simplified the implementation by a lot; the IdentityStore for bearer tokens is no longer needed. I still use the fish.payara.security.openid.api.AccessTokenCallerPrincipal class, so the fish dependency could not be removed yet. Note, however, that the implementation got very short, it should not be a problem to replace it with some other implementation in the future, if needed.

I am still not sure about the auth filters thing. It looks nice to me how it is now, it simply uses SecurityContext added to the abstract API bean as context. Also, are we going to need all of those filters when we will migrate to the new SPA? Maybe they will become deprecated?

The handling of the user IDs got also better. The issuer and subject IDs used to create the UserIdentifier are configurable now. This would solve our problem, where we would like to differentiate between different providers configured in one broker (one Keycloak with only one url), where you would only get edit rights when you are authenticated with our internal two factor authentication. You can also change where the subject name is retrieved from, we could, e.g., use the "preferred_email" as configured in our Keycloak, i.s.o. the sub field, which would make mapping of the users easier (we preprocess the users, such that the internal KU Leuven users are known by the Dataverse before they login for the first time).

ErykKul · 2024-10-15T20:59:03Z

Oh no! I looked at how the communication is done with Keycloak and it does not look good! It is full of JWT tokens in cookies and even a session cookie! (this might be a security risk, would need to investigate more, I hope these can be turned off):

ErykKul · 2024-10-16T08:03:11Z

If we do not want the Payara implementation, there are some alternatives. For example, there is a Spring Boot version that also uses the SecurityContext, it looks like it would not be difficult to switch to that. There are also plenty of other possibilities, we really do not have to implement everything ourselves (and all implementations look very similar, as I mentioned in the tech hours, and most likely they use sessions and cookies, like Keycloak does, etc.):

https://docs.spring.io/spring-security/reference/servlet/oauth2/index.html

ErykKul · 2024-10-17T11:52:19Z

The more I look into it, the more I like the solution from this PR using Payara builtin OIDC support. It integrates very nicely (yes, it can be still improved, but it works very well already) with Dataverse and allows us to use security annotations in the future to secure specific endpoints, without any complex implementations of authentication filters or getUserOrDie. Also, the existing authentication mechanisms we might want to keep, like maybe the URL-signing, can be easily implemented by implementing the IdentityStore interface. All of the functionality we might need in the future can be added via additional middleware. For example, the support for Opaque tokens that is dockerized and exchanges Opaque tokens to JWT tokens is readily available:

https://curity.io/resources/learn/integration-kong-open-source/

If an institution only has Shibboleth, and does not want to have a Keycloak, they can just drop a plugin in their Shibboleth:

https://shibboleth.atlassian.net/wiki/spaces/IDPPLUGINS/pages/1376878976/OIDC+OP

You want a builtin user support?

implement username/password HTML form.

If we would like to migrate to a different server than Payara in the future, pure Java EE is not sufficient (it is most likely the case for more use-cases than security only), we would still need some additional middleware, like already mentioned Spring Boot. We could also use some other middleware, maybe for the security only. If we go for micro-service architecture, a whole world of possibilities opens up, we could use something like the OAuth2-proxy from my other PR.

If I am not mistaken, all of the OAuth2 providers we support now, like ORCID, support OIDC already, such that the things could be simplified even more.

edit:

I found also this about the Opaque tokens:

"The extraParameters param is used to send the audience value as the Auth0 custom API (without which, Auth0 will return an opaque token). "

and this:

"To set this using Microprofile Config use {@code payara.security.openid.extraParams.raw}, in URL query format:
{@code key=value&key2=value+with+spaces}. The keys may repeat."

So, it looks like the Opaque tokens can be avoided.

github-actions · 2024-11-18T14:38:47Z

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:authn-arch-api-oidc

ghcr.io/gdcc/configbaker:authn-arch-api-oidc

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

api oidc authentication mechanism

81d0ee0

ErykKul requested a review from pdurbin October 3, 2024 15:33

github-actions bot reviewed Oct 3, 2024

View reviewed changes

src/main/java/edu/harvard/iq/dataverse/api/OpenIDCallback.java Outdated Show resolved Hide resolved

replaced tabs with spaces

96fab76

pdurbin changed the title ~~api oidc authentication mechanism~~ API OIDC authentication mechanism Oct 3, 2024

pdurbin requested review from johannes-darms and poikilotherm October 3, 2024 15:46

pdurbin self-assigned this Oct 3, 2024

pdurbin added the Size: 10 A percentage of a sprint. 7 hours. label Oct 3, 2024