feat: Add resource-info-fetcher by sbernauer · Pull Request #863 · stackabletech/opa-operator

sbernauer · 2026-07-21T08:49:34Z

Description

Part of #848

Yes, the diff looks big, but most of it are lockfiles and kuttl tests

CRD change

Basically the same as for the user-info-fetcher. The only difference is the env field, which is DataHub specific.

Look at extra/crds.yaml for the concrete CRD change

  clusterConfig:
    resourceInfo: # <1>
      backend:
        dataHub:
          hostname: datahub-gms.my-namespace.svc.cluster.local
          port: 8080
          rootPath: / # optional, defaults to /
          tls:
            verification:
              server:
                caCert:
                  secretClass: tls # <2>
          credentialsSecretName: resource-info-fetcher-credentials # <3>
          env: PROD # <4>
      cache: # optional, enabled by default
        maxEntries: 42 # optional, defaults to no limit
        entryTimeToLive: 60s # optional, defaults to 60s
---
apiVersion: v1
kind: Secret
metadata:
  name: resource-info-fetcher-credentials
stringData:
  token: <datahub-personal-access-token> # <3>

API of RIF

The rego rules and HTTP API of RIF: https://github.com/stackabletech/opa-operator/blob/spike/rif/docs/modules/opa/pages/usage-guide/resource-info-fetcher.adoc

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
CRD changes approved
CRD documentation for all fields, following the style guide.
Helm chart can be installed and deployed operator works
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible
Links to generated (nightly) docs added
Release note snippet added

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added
Links to generated (nightly) docs added
Release note snippet added
Add type/deprecation label & add to the deprecation schedule
Add type/experimental label & add to the experimental features tracker

Release notes

Added

Add an initial version of resource-info-fetcher, which is similar to user-info-fetcher, but allows to fetch additional metadata about resource information from a data catalog.
For now only DataHub is supported.
Also, a rego-rule library has been added to make it easier to call resource-info-fetcher from within OPA.
The API (especially the response) might change in the future once more data catalogs are supported

Nightly docs

As soon as this PR is merged the docs will show up at https://docs.stackable.tech/home/nightly/opa/usage-guide/resource-info-fetcher/

lfrancke · 2026-07-30T15:03:29Z

Because we had it yesterday with OpenLineage: I believe it could make sense to add a "path" or "endpoint" field. Just in case DataHub is deployed behind some proxy. ://

sbernauer · 2026-07-31T07:31:35Z

I believe it could make sense to add a "path" or "endpoint" field

Good point! For consistency reasons with UIF and AuthenticationClass I went with rootPath in 215d4b9. Feel free to have a look

Maleware

Just two typos, thanks you! Test passes locally, thus LTGTM otherwise.

Maleware · 2026-08-04T17:14:19Z

+}
+
+/// Generates the trivial `From<Params> for ResourceInfoRequest` conversions, so each HTTP handler
+/// can turn its deserialized query parameters into a [`ResourceInfoRequest`] via `.into()`. Adding a


Suggested change

/// can turn its deserialized query parameters into a [`ResourceInfoRequest`] via `.into()`. Adding a

/// can turn its deserialized query parameters into a [`ResourceInfoRequest`] via `.from()`. Adding a

https://doc.rust-lang.org/std/convert/trait.From.html
From automatically implements Into, so the caller can pick.
In

opa-operator/rust/resource-info-fetcher/src/main.rs

Line 185 in 970da87

P: DeserializeOwned + Into<ResourceInfoRequest>,

we actually use Into<ResourceInfoRequest>. In general I feel in this situation a into feels more natural than a from

Ah okay allright. I was looking at the function beneath which reads from thought that might want to match.

Co-authored-by: Maximilian Wittich <56642549+Maleware@users.noreply.github.com>

sbernauer · 2026-08-05T05:59:27Z

Please vote on the CRD change as well as the rego rule API on this comment. Feel free to also look at the JSON API, but I'd say that's an implementation detail we are allowed to change in the future

maltesander

First round, did not check docs etc. properly yet and not testing yet.

maltesander · 2026-08-05T07:45:45Z

+        // Trim trailing whitespace/newlines so the value is safe to use in an HTTP header.
+        let token = tokio::fs::read_to_string(&token_path)


This is never refreshed? Will go stale and produce 401 until restart?

Yes, but ResolvedKeycloakBackend::resolve, ResolvedEntraBackend::resolve, ResolvedOpenLdapBackend::resolve and maybe others do the exact same thing.
According to https://docs.datahub.com/docs/authentication/personal-access-tokens you can create a PAT without expiration.
I see that is not 100% ideal, but also IMHO the status quo basically everywhere else in the platform as well

No i dont think so. They read the client credentials, not the token?

opa-operator/rust/user-info-fetcher/src/backend/keycloak.rs

Line 189 in 8402d62

let authn = send_json_request::<OAuthResponse>(

(same for entra).
This is not ideal either, we should just cache it and refresh on 401.

maltesander · 2026-08-05T08:16:35Z

+///
+/// [`urn_for_request`]: crate::backend::data_hub::resource_to_urn_mapping::urn_for_request
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub enum ResourceInfoRequest {


The instance is always required. Datahub (defaults) leave that out and a trino table will become e.g. urn:li:dataset:(urn:li:dataPlatform:trino,tpch.sf1.customer,PROD) and the tableResourceInfo always builds {instance}.tpch.sf1.customer which will be empty and therefore fail a correct rego?

Can we make instance optional and skip?

I was thinking of this as well (I actually started without the instance).
I think we should very strongly recommend users to set the instance when ingesting data into DataHub.
Also we likely want to support other backends in the future.
If they don't have such a strange concept of "there is only one Trino" they will require us to send a instance for every request.
So I fear we would have a problem with future backends if we make it optional.

Hmm i dont think we should strongly recommend how people use a non stackable product?

I think the scope of this PR is datahub, so i would try to represent everything datahub can do with reasonable abstraction? Removing the optional part can then be done once another backend lands?

I feel this is a too generalized and therefore cutting into datahub funtionality?

I also had to refactor the crd mod to be only pulled in via lib.rs to fix clippy warnings

maltesander

Claude reproduced a bigger security issue (policies can pass if backend is gone).

Silent fail-open when the backend is unavailable

Backend state	`allow if tag==public`	`deny if tag==pii`
healthy	deny	deny ✅
PAT rejected (401)	deny	allow ❌
GMS scaled to 0	deny	allow ❌

All three containers report ready=true throughout. There is no readiness probe on the sidecar.

Repro: kubectl -n $NS scale deploy datahub-datahub-gms --replicas=0, wait 60s for the cache to drain, query any policy that excludes a tag.
Fix: add a readiness probe in controller/build/resource/daemonset/resource_info_fetcher.rs

Tags and domains are not inherited from the parent container

Tagging schema tpch.sf1 with pii leaves child table nation reporting tags:[public], domain:null. A deny if pii policy allows it. DataHub exposes the parent in the same query (container { tags domain }), so this is one field away.

Repro: addTag/setDomain on the schema container URN, wait out the 60 s TTL, query the table.
Fix: add container { tags {...Tags} domain {...Domain} } to the fragment at graphql.rs:30, or state the non-inheritance explicitly in the docs.

maltesander · 2026-08-11T06:32:56Z

+            Self::ConstructHttpClient { .. } => StatusCode::SERVICE_UNAVAILABLE,
+            Self::BuildDataHubEndpoint { .. } => StatusCode::BAD_REQUEST,
+            Self::ExecuteGraphQlQuery { .. } => StatusCode::INTERNAL_SERVER_ERROR,
+            Self::GraphQlErrors { .. } => StatusCode::INTERNAL_SERVER_ERROR,


I think this is user-controlled, e.g. a bad identifier is rather a client error?

Suggested change

Self::GraphQlErrors { .. } => StatusCode::INTERNAL_SERVER_ERROR,

Self::GraphQlErrors { .. } => StatusCode::BAD_REQUEST,

sbernauer added 28 commits July 16, 2026 10:18

Initial development of resource-info-fetcher

ccadaf8

feat: Implement Trino catalog and schema

a1da9db

feat: Cache tag and user lookups

d56a198

feat: Lookup Superset things (by id)

f28fdb7

chore: More parallism

90b010d

feat: Add Kafka topic support

53e83c6

Crying in GraphQL

0e2857a

feat: Also fetch domains and data products

551c2f6

RawDataHubUrn -> DataHubUrn

a8310f6

refactor: Move user-info-fetcher sidecar creation into separate module

a25da2b

Merge remote-tracking branch 'origin/main' into spike/rif

e5d1f05

Merge branch 'refactor/uif-sidecar' into spike/rif

9698dc8

Re-add Tiltfile

f5b5bf9

Implement RIF into operator

2cfd78d

Improve test

86127a6

chore: Switch endpoints from complex enum to path based

554a7f2

chore: Move env out of global config to the requests

e5ff314

WIP for Maxi :)

970da87

fix kuttl test

2e4655f

refactor Python assert script

8c9d8c5

refactor Python script

dff0f99

Merge remote-tracking branch 'origin/main' into spike/rif

74c648d

Remove leftover files

32cd6e2

cargo fmt

9d95f1f

Fix OpenShift on high disk usage

d52110c

Merge remote-tracking branch 'origin/main' into spike/rif

26f51e9

cargo update

7c11523

Slightly improve docs

fc889ca

sbernauer marked this pull request as ready for review July 30, 2026 13:49

Stupid YAML linter

59af952

feat: Add rootPath field for DataHub

215d4b9

changelog

4dc05fc

sbernauer self-assigned this Jul 31, 2026

sbernauer added 2 commits July 31, 2026 09:49

Improve code comments

6571afa

regenerate nix

f770b04

sbernauer requested a review from Maleware July 31, 2026 08:53

sbernauer added this to Stackable Engineering Jul 31, 2026

sbernauer moved this to Development: Waiting for Review in Stackable Engineering Jul 31, 2026

Maleware reviewed Aug 4, 2026

View reviewed changes

Update rust/resource-info-fetcher/src/main.rs

8c2a0df

Co-authored-by: Maximilian Wittich <56642549+Maleware@users.noreply.github.com>

sbernauer requested a review from Maleware August 5, 2026 05:56

Merge branch 'main' into spike/rif

af16a08

maltesander self-requested a review August 5, 2026 06:26

maltesander requested changes Aug 5, 2026

View reviewed changes

sbernauer added 2 commits August 6, 2026 16:26

docs: Link in nav

bf85e53

WIP: Error when too many data products are returned

cd9a38e

sbernauer moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Aug 10, 2026

sbernauer added 5 commits August 10, 2026 13:22

Merge remote-tracking branch 'origin/main' into spike/rif

fde8854

fix: Use utils::http::client_builder helper

c20d1d1

clippy

e3e84ed

feat: Allow configuring max cache size.

f481d44

I also had to refactor the crd mod to be only pulled in via lib.rs to fix clippy warnings

regenerate nix

fdcb012

sbernauer requested a review from maltesander August 10, 2026 12:56

maltesander reviewed Aug 11, 2026

View reviewed changes

Maleware mentioned this pull request Aug 11, 2026

Feature: Allow workloadKind: Deployment #873

Draft

20 tasks

	/// can turn its deserialized query parameters into a [`ResourceInfoRequest`] via `.into()`. Adding a
	/// can turn its deserialized query parameters into a [`ResourceInfoRequest`] via `.from()`. Adding a

		// Trim trailing whitespace/newlines so the value is safe to use in an HTTP header.
		let token = tokio::fs::read_to_string(&token_path)

	Self::GraphQlErrors { .. } => StatusCode::INTERNAL_SERVER_ERROR,
	Self::GraphQlErrors { .. } => StatusCode::BAD_REQUEST,

Uh oh!

Uh oh!

Conversation

sbernauer commented Jul 21, 2026 • edited by maltesander Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

CRD change

API of RIF

Definition of Done Checklist

Author

Reviewer

Acceptance

Release notes

Added

Nightly docs

Uh oh!

lfrancke commented Jul 30, 2026

Uh oh!

sbernauer commented Jul 31, 2026

Uh oh!

Maleware left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sbernauer commented Aug 5, 2026

Uh oh!

maltesander left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sbernauer Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maltesander left a comment

Choose a reason for hiding this comment

Silent fail-open when the backend is unavailable

Tags and domains are not inherited from the parent container

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sbernauer commented Jul 21, 2026 •

edited by maltesander

Loading

sbernauer Aug 10, 2026 •

edited

Loading