Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for externalized consumers #7657

Open
wants to merge 18 commits into
base: main
Choose a base branch
from

Conversation

jschmid1
Copy link
Contributor

@jschmid1 jschmid1 commented Jul 18, 2024

Copy link

netlify bot commented Jul 18, 2024

Deploy Preview for kongdocs ready!

Name Link
🔨 Latest commit a3f84e0
🔍 Latest deploy log https://app.netlify.com/sites/kongdocs/deploys/67dc2574cdfac00008637459
😎 Deploy Preview https://deploy-preview-7657--kongdocs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
Lighthouse
Lighthouse
9 paths audited
Performance: 92 (🟢 up 1 from production)
Accessibility: 93 (no change from production)
Best Practices: 98 (🟢 up 8 from production)
SEO: 99 (no change from production)
PWA: -
View the detailed breakdown and full score reports

To edit notification comments on pull requests, go to your Netlify site configuration.

@jschmid1 jschmid1 added the do not merge Issues/ PRs whose changes should not be merged at this time label Jul 18, 2024
@Guaris Guaris added the review:general Review for general accuracy and presentation. Does the doc work? Does it output correctly? label Jul 22, 2024
@Guaris Guaris requested a review from lena-larionova July 22, 2024 16:12
@lena-larionova lena-larionova added this to the Gateway 3.8 milestone Jul 23, 2024
@Guaris Guaris self-assigned this Aug 19, 2024
@Guaris Guaris marked this pull request as ready for review August 19, 2024 16:29
@Guaris Guaris requested a review from a team as a code owner August 19, 2024 16:29
Comment on lines 366 to 367
- geo: null
id: null
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

geo and id should be optional (default null) imo


-- Dummy/Link that points to Konnect Docs when ready.

With the `pool_id` you obtained from the previous step, you can configure the key-auth plugin to validate API keys against the Identity Service.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jschmid1 where does the pool_id come from? Konnect?

Copy link
Contributor

@smritikjaggi smritikjaggi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to change all references of pools to realms, and update the field terminology

@@ -0,0 +1,71 @@
---
nav_title:
title: How to configure pools
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to Realms

---


With `pool_id` you can configure the key-auth plugin to validate API keys against the Identity Service.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to "realm_id"


With `pool_id` you can configure the key-auth plugin to validate API keys against the Identity Service.

### Configuring Multiple Pools
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to "Realms"


### Configuring Multiple Pools

In the key-auth plugin configuration, add the `pools` option as shown below:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to "realms"

In the key-auth plugin configuration, add the `pools` option as shown below:

```yaml
pools:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to "identity_realms"

type: remote
```

In this configuration, the dataplane will initially check the local pool (LMDB) before querying the remote Identity Service.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cp-scoped consumers first


In this configuration, the dataplane will initially check the local pool (LMDB) before querying the remote Identity Service.

If a matching key is found in any of these pools, the request will be authenticated. If the key is not found in any of the configured pools, the request will be blocked.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"realms"


If a matching key is found in any of these pools, the request will be authenticated. If the key is not found in any of the configured pools, the request will be blocked.

### Configuring Single Pools
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Single Realm


### Configuring Single Pools

It is also possible to configure only a single pool, either local or remote. However, only one of each type can be configured.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change pool to realm
change "either local or remote" to "consumers can be scoped to the geo or cp".


It is also possible to configure only a single pool, either local or remote. However, only one of each type can be configured.

To configure only a remote pool:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change "remote pool" to "realm"

@lena-larionova lena-larionova removed this from the Gateway 3.8 milestone Sep 10, 2024
smritikjaggi and others added 4 commits March 6, 2025 11:49
Testing edit function
Updated the terminology used for realms and the yaml file examples
@cloudjumpercat
Copy link
Contributor

cloudjumpercat commented Mar 11, 2025

@brentos Hi! I just wanted to let you know I'll be working on external consumers and I just added some changes to the PR, including a new Konnect page that explains what they are and how to configure them, as well as a revised section to key auth. I had a few questions while I was working on it:

  • We mention that a kong dev build is required, but this is for Konnect. I don't understand how that custom dev build is required unless we're using it for data plane nodes?
  • I tried using the /realms endpoint, but wasn't able to access it. Maybe something to do with not having the custom build?
  • Are we publishing the API specs for this release? Can I have a copy of them so I can look at the endpoints and parameters?
  • So key auth only associates the plugin with a realm, it doesn't actually create the realm, you use the /realm endpoint to do that?

Signed-off-by: Diana <[email protected]>
Signed-off-by: Diana <[email protected]>
@cloudjumpercat
Copy link
Contributor

Broken sales link was a false failure, I manually approved link validation for now

@cloudjumpercat
Copy link
Contributor


The following diagram shows how you can configure centralized Consumers in realms as well as scope Consumers to a Control Plane:

<!--vale off -->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand what this diagram is showing me.
For example:

Consumer: Ira
Consumer Groups: Gold for appointments

Does this consumer only have access to the appointments control plane? Or does it have access to both control planes, but since the consumer group Gold only exists on appointments, it can only access appointments? Or something else?

Are control planes located inside realms, or alongside them?

* The Consumer identity can be setup centrally instead of being defined across multiple control planes.
* Share Consumers across multiple control planes. Users don't need to replicate changes to Consumer identity in multiple control planes and Consumer configuration doesn't conflict.
* Reduces configuration sync issues between the control plane and the dataplane. Centralized Consumers aren't part of the configuration that is pushed down from the control plane to the dataplane, so it reduces config size and latency.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add the required Gateway version here as a note, somewhere.

Comment on lines 338 to 341
* **Realm is listed first:** The dataplane will first reach out to the realm. If the API key is not found in the realm, the dataplane will look for the API key in the control plane config.
* **Control Plane scope listed first:** The dataplane will initially check the control plane configuration (LMDB) for the API key before looking up the API Key in the realm.
* **Realm isn't associated with a control plane:** You can also configure a single `identity_realms` by omitting the `scope: cp` from the example. In this case, the dataplane will only attempt to authenticate API keys against the realm. If the API key isn't found, the request will be blocked.
* **Realm is only associated with a Control Plane:** You can configure a look up only in the control plane config by only specifying `scope: cp` for `identity_realms`. In this scenario, the dataplane will only check the control plane configuration (LMDB) for API key authentication. If the API key isn't found, the request will be blocked.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* **Realm is listed first:** The dataplane will first reach out to the realm. If the API key is not found in the realm, the dataplane will look for the API key in the control plane config.
* **Control Plane scope listed first:** The dataplane will initially check the control plane configuration (LMDB) for the API key before looking up the API Key in the realm.
* **Realm isn't associated with a control plane:** You can also configure a single `identity_realms` by omitting the `scope: cp` from the example. In this case, the dataplane will only attempt to authenticate API keys against the realm. If the API key isn't found, the request will be blocked.
* **Realm is only associated with a Control Plane:** You can configure a look up only in the control plane config by only specifying `scope: cp` for `identity_realms`. In this scenario, the dataplane will only check the control plane configuration (LMDB) for API key authentication. If the API key isn't found, the request will be blocked.
* **Realm is listed first:** The data plane will first reach out to the realm. If the API key is not found in the realm, the data plane will look for the API key in the control plane config.
* **Control Plane scope listed first:** The data plane will initially check the control plane configuration (LMDB) for the API key before looking up the API Key in the realm.
* **Realm isn't associated with a control plane:** You can also configure a single `identity_realms` by omitting the `scope: cp` from the example. In this case, the data plane will only attempt to authenticate API keys against the realm. If the API key isn't found, the request will be blocked.
* **Realm is only associated with a Control Plane:** You can configure a look up only in the control plane config by only specifying `scope: cp` for `identity_realms`. In this scenario, the data plane will only check the control plane configuration (LMDB) for API key authentication. If the API key isn't found, the request will be blocked.

Should this info actually be on the Konnect page instead, except genericized to "credentials" instead of "API key"?

<!--vale off -->
{% mermaid %}
flowchart TD
A(Consumer: Cruz<br>Consumer Groups: Gold for appointments, Silver for billing)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each consumer can be in multiple consumer groups, but you can't scope the consumer group to a service.

If billing respected the Gold group, then Cruz would be Gold for billing too.

* {region}: Region for your {{site.konnect_short_name}} instance.
* $KONNECT_TOKEN: Replace with your {{site.konnect_short_name}} personal access token.
* $CONTROL_PLANE_UUID: (Optional) Replace with your Control Plane UUID.
* `ttl` and `negative_ttl`: (Optional) Time-to-live in minutes of the consumer or negative consumer for this realm in the {{site.base_gateway}} cache.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's a negative consumer? I don't think that this is what negative_ttl means. @zekth can you help us out here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's no concept of negative_consumer. negative_ttl represents the TTL of a bad login cache entry. Meaning api key foobar is sent to the gateway; gateway reaches the service to authenticate it; if it fails the api key foobar will be labelled as non working in gateway cache to the time of the TTL. During that time it won't try to reache the authentication server.

* $KONNECT_TOKEN: Replace with your {{site.konnect_short_name}} personal access token.
* {realmId}: The ID of the realm you created previously.
* $CONSUMER_NAME: Replace with the name of the consumer.
* $CONSUMER_GROUP: (Optional) Replace with the name of the consumer groups you want to associate with the consumer.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I specify consumer_groups on a consumer does that replace consumer_groups from the realm, or are they added together?

If I specify consumer_groups: [] does that mean no consumer groups, or use the groups from the realm?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is additive here, I'll add that info to the doc.

Comment on lines 340 to 341
* **Realm isn't associated with a control plane:** You can also configure a single `identity_realms` by omitting the `scope: cp` from the example. In this case, the dataplane will only attempt to authenticate API keys against the realm. If the API key isn't found, the request will be blocked.
* **Realm is only associated with a Control Plane:** You can configure a look up only in the control plane config by only specifying `scope: cp` for `identity_realms`. In this scenario, the dataplane will only check the control plane configuration (LMDB) for API key authentication. If the API key isn't found, the request will be blocked.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* **Realm isn't associated with a control plane:** You can also configure a single `identity_realms` by omitting the `scope: cp` from the example. In this case, the dataplane will only attempt to authenticate API keys against the realm. If the API key isn't found, the request will be blocked.
* **Realm is only associated with a Control Plane:** You can configure a look up only in the control plane config by only specifying `scope: cp` for `identity_realms`. In this scenario, the dataplane will only check the control plane configuration (LMDB) for API key authentication. If the API key isn't found, the request will be blocked.
* **Realm only:** You can also configure a single `identity_realms` by omitting the `scope: cp` from the example. In this case, the dataplane will only attempt to authenticate API keys against the realm. If the API key isn't found, the request will be blocked.
* **Control Plane only:** You can configure a look up only in the control plane config by only specifying `scope: cp` for `identity_realms`. In this scenario, the dataplane will only check the control plane configuration (LMDB) for API key authentication. If the API key isn't found, the request will be blocked.

@@ -287,19 +295,55 @@ Response:
}
```

{% if_version gte: 3.10.x %}

## Configure realms for centralized Consumers in {{site.konnect_short_name}}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be an example rather than on the overview?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I'll convert it to an example.

cloudjumpercat and others added 2 commits March 14, 2025 10:00
Copy link
Contributor

@smritikjaggi smritikjaggi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The content looks good Diana! Can we talk through the naming and placement of Consumers please? This doc outlines that we decided to go with Consumers as the official naming of this initiative: https://docs.google.com/document/d/1k-d9FqAk4Jqo-e9-19IekqKQlNWfaFdgBNBuJGBw80k/edit?tab=t.0#heading=h.tn4g33ltt5z7

@@ -15,6 +15,8 @@
url: /private-connections/aws-privatelink/
- text: Geographic Regions
url: /geo/
- text: Regional Consumers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are going with the naming of "Consumers" as we are not introducing a new entity, but are expanding on the existing Consumer entity such that they can be defined in a central location per region and referenced across multiple control planes. Doc: https://docs.google.com/document/d/1k-d9FqAk4Jqo-e9-19IekqKQlNWfaFdgBNBuJGBw80k/edit?tab=t.0#heading=h.tn4g33ltt5z7

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good! I'll revise this to call out that this is a new type of consumer management.

title: Configure realms for regional Consumers in {{site.konnect_short_name}}
---

Starting in {{site.base_gateway]] 3.10.x, credentials that are stored centrally in {{site.konnect_short_name}} that are shared across multiple {{site.base_gateway}} can be validated by configuring `identity_realms` field in the Key Auth plugin. You can use these realms when you create [regional Consumers](/konnect/regional-consumers/).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to add some language that says "today a DP can only reach out to realms in the same region as they are deployed"?

@@ -0,0 +1,78 @@
---
title: Regional Consumers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our official name for this initiative is "Consumers". What is the best way for us to reflect that this initiative is an expansion of the existing consumer entity such that consumers can be stored in a centralized space per region?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my revision, I'll call them "centralized consumer management" or "centrally managed consumers" if I need to call out the difference between them and "original" consumers. Otherwise, I'll use "consumers" when the context makes it clear we're talking about the central ones.

* `$CONSUMER_GROUP`: (Optional) Replace with the name of the consumer groups you want to associate with the consumer. Consumer groups set here are additive. This means that if you configure the realm with consumer groups A and B, and then configure the regional consumer with consumer group C, the authenticated consumer will be assigned to consumer groups A, B, and C.
1. Consumers require authentication. Configure authentication using the [Key Auth plugin](/hub/kong-inc/key-auth/how-to/).

The order in which you configure the `identity_realms` dictates the priority in which the data plane attempts to authenticate the provided API keys:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can also list out that the default is scope:cp for Key Auth plugin.

Copy link
Contributor

@smritikjaggi smritikjaggi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Left some comments

Save the ID of the realm.

Be sure to replace the following with your own values:
* `{region}`: Region for your {{site.konnect_short_name}} instance.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Today DPs can only reach out to the realms defined in the same region as the DP. Should we add this info somewhere?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, I can add that somewhere!

{:.important}
> Centralized consumer management is only available for {{site.base_gateway}} 3.10 data planes. Make sure your control plane uses 3.10 data planes at minimum when configuring these consumers.

1. Use the `/realms` endpoint to create a realm and optionally associate it with allowed control planes and time-to-live values:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We define time-to-live further down the document. Should we define it here as well?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll leave this one as-is since we typically follow this format in docs where we explain it later in the bullets and it could bloat the step instruction if I put it here.


{:.note}
> **Note:** If you are using KIC to manage your data plane nodes in {{site.konnect_short_name}}, ensure that you configure the `telemetry_endpoint` in the data plane. You can find the `telemetry_endpoint` in the {{site.konnect_short_name}} UI in [Gateway Manager](https://cloud.konghq.com/gateway-manager/) in the data plane node instructions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to add a section around changing the scope of consumers from cp to realm? The user will need to do the following call:

POST https://us.api.konghq.com/identity/pools/%7BpoolID%7D/consumers)/Alice-123/keys
{
"type": "legacy" // Migrating Alice's existing keys
"secret": "Alice-api-key"
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since that's optional, I'd rather link them to the API spec so they can see what else they can do. Do you have a predicted link to the public API spec?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am adding a new step to create new keys for the consumer, since I think that was missing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci:manual-approve:link-validation do not merge Issues/ PRs whose changes should not be merged at this time review:general Review for general accuracy and presentation. Does the doc work? Does it output correctly?
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants