API-Key generation API is missing

[SebastianGode]

Is your feature request related to a problem? Please describe.
Instead of using normal user+password authentication I would like to have an never expiring API key as Elasticsearch also offers.
https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-api-create-api-key.html
Currently I have not found any way to generate such a key as the security plugin does not have such an option yet:
https://opensearch.org/docs/latest/security-plugin/access-control/api/

Describe the solution you'd like
It would be great to have an Elasticsearch compatible API for generating API-keys

[jcannell]

Was just looking for this feature today...would love to have this.

[maddisondavid]

+1 I was also searching around thinking I'd missed something in the documentation!

[feli0821]

+1 also looking for this feature

[jansyren]

+1 yes this would be a great feature, lots of applications writing to elastic has this support so adding it to Opensearch would be a good improvement.

[djfranzelttec]

Would be an excellent feature to have.

[emollusion]

+1 on this one, good to have when implementing "machine to machine" where cert auth is not possible.

[bidetzz]

+1 this feature would be a great addition

[nundys]

+1

[derek-ho]

Requirements

API tokens would be useful in several situations, including user and service to service authentication/authorization. Once available in the OpenSearch security ecosystem, API tokens can also be used as the foundational building blocks for other features such as enforcing rate or resource usage limits to asynchronous/long running jobs.

Based on the above the main user stories we are trying to solve would be the following:

• As a user, if I have permissions to do so, I should be able to create an API token (with a subset of my permissions?) for a specific amount of time (or unlimited) that can be used for authc/authz.
  • These API tokens should be able to perform actions that fall under the given permissions at creation time for the specified lifetime of the api token, and there should be some mechanism to track these requests (for future enhancements).
• As a user, if I have permissions to do so, I should be able to list the currently existing and revoked API tokens.
  • It would be a nice to have to be able to list the last used date/time of tokens
• As a user, if I have permissions to do so, I should be able to delete/revoke a valid API token. After doing so, that token should immediately not be able to be used for authc/authz.

Design

Approach 1: Index a Document for each Token, AllowList Cache for Usage (Preferred)

Components:

• Index .opensearch_security_api_tokens (different index for different settings/policies from opendistro_security)

  • Document structure:

      JTI: string
      revoked_at: int (epoch time of revocation)
      last_used: int (epoch time of last usage)
      status: active | revoked | deleted
      } 
  

• config.yml :

              description: "Authenticate via API Token"
              http_enabled: false
              transport_enabled: false
              order: 0
              http_authenticator:
                type: apitoken
                challenge: false
                config:
                  signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
              authentication_backend:
                type: noop 
  
  
  

• API Token allow cache size limit: 100 (subject to change)

• Number of API Tokens outstanding at any time limit: 100 (subject to change)

Create token flow:

POST /apitokens

{
   des: string (required), same restrictions/validations as username
   exp: int (optional), epoch time, null means never expires 
   permissions: {
    "cluster_permissions": [
        "cluster_manage_pipelines",
        "cluster_all"
    ],
    "index_permissions": [
        {
            "index_patterns": [
                "t"
            ],
            "dls": "",
            "fls": [],
            "masked_fields": [],
            "allowed_actions": [
                "data_access"
            ]
        }
    ],
    "tenant_permissions": []
}
}

• When the user sends a request to the create API token endpoint, we first check that the user is an admin (cert) or has restapi:admin/ratelimiters permissions
  • If they don’t reject the request with 401
  • If the requested permissions of the token is not a subset of the user’s permissions, also reject the request with a 400
• If they do, we use the signing_key to create a JWT that encodes all of the fields that was sent during the creation request and return it to the user. In the corresponding index .opensearch_security_api_tokens , we create a document with the jti of the newly created api token.
• We also add the JTI into the cached allowlist of API tokens

Revoke API Token flow

• When the user sends a request to the revoke an api token, we first check that the user is an admin (cert) or has restapi:admin/ratelimiters permissions
  • If they don’t reject the request with 401
• If they do, we lookup the corresponding document in the .opensearch_security_api_tokens system index, and update the revoked_at field accordingly
• We also add evict the JTI of this token from the cached allowlist of API tokens

Delete API Token flow

• When the user sends a request to delete an api token, we first check that the user is an admin (cert) or has restapi:admin/ratelimiters permissions
  • If they don’t reject the request with 401
• IF they do, we mark the corresponding document in the .opensearch_security_api_tokens system index as deleted
• We also evict the JTI of this token from the cached allowlist of API tokens

Get All API Token flow

• When the user sends a request to list all api token, we first check that the user is an admin (cert) or has restapi:admin/ratelimiters permissions
  • If they don’t reject the request with 401
• If they do, we first run a job that syncs up the last usage time of each JTI from the cache to the index
• we list all documents in the .opensearch_security_api_tokens system index, along with their revoked_at times, if applicable

Use API Token for Authc/AuthZ flow

• We add a new type of HTTP Authenticator called apitoken . This will be a authC domain, similar to JWT today, with corresponding evaluation order. The necessary config here is the signing key, which we will use to create/verify the token sent when using API tokens for authc/authz
• When the user sends a request with the api token given them on creation, we first hash it and verify that the JTI exists in the allowlist cache.
  • If it doesn’t, we reject the request with a 401
  • If it does, we create a role in memory based on the contents of the api token, and use this to authorize the request.
    • We also update the last usage of the token to the current time/date
    • We will run a scheduled job to flush this information to the index

Key Concerns/Considerations

• Two options for how to authorize api token requests: perform an index lookup on every request to check validity of token, or use a cache to track whether a JTI is valid or not. Preferred option is to use a cache.
  • Concern 1: Cache grows too large resulting in increased heap usage.
    • Users must have API permissions to create/revoke API tokens in the first place, which would be limited to admins + trusted individuals. Having a restriction on the number of valid API tokens outstanding should also negate this concern.
  • Concern 2: Cache could overflow/get out of sync with the index, leading to unauthorized requests going through (for example using revoked API tokens).
    • We have an index listener on the system index which would add all active JTIs into the cache on any index changes.
  • Concern 3: Cache lost during cluster upgrade/ down time
    • On security plugin initialization we sync the contents of the index into the cache, so any upgrades or downtime should be accounted for

Approach 2: Index a document for each token, DenyList Cache

• Same general idea as Approach 1, but instead of only allowing api tokens to be used if they exist in the allowlist cache, we instead assume all well-structured tokens are valid, except for those that we explicitly store in a deny list cache.
• No tangible benefits over allowlist cache
• The main drawback here is that the number of revoked/deleted tokens can go to infinity, but the size of the cache is finite and could affect the cluster performance. Therefore, it makes more sense to me to track something in the cache that is finite.

Approach 3: Do an index lookup for each request to verify validity of api token

• Instead of using a cache to figure out if tokens can be used, we instead perform an index lookup on each request to verify that a token can be used
• Pros: simplifies the overall design, a cache is no longer needed
• Cons: the overhead for performing an index lookup for each request may be too large, leading to infeasibility of this design (should we get some numbers to support this?)

[derek-ho]

Task breakdown / sub tasks:

• Initialize api tokens index if it doesn't exist  #4899

• [milestone] Create a cache that listens in on API tokens index if feature is enabled #4902

• [milestone] Create API Token flow #4903

• [milestone] Revoke API Token Flow #4904

• [milestone] Use API Token Flow #4905

Working on feature branch: https://github.com/opensearch-project/security/tree/feature/api-tokens