diff --git a/README.md b/README.md index 0038314..0bcf824 100644 --- a/README.md +++ b/README.md @@ -275,6 +275,7 @@ spec: | `servicePrincipalsRef` | string | Reference to resolve a list of service principal names from `spec`, `status` or `context` (e.g., `spec.servicePrincipalConfig.names`) | | `target` | string | Required. Where to store the query results. Can be `status.` or `context.` | | `skipQueryWhenTargetHasData` | bool | Optional. When true, will skip the query if the target already has data | +| `queryInterval` | string | Optional. Minimum interval between queries as a Go duration string (e.g. `10m`, `1h`, `90s`). Skips querying Microsoft Graph until the interval has elapsed since the last successful query, independent of reconcile frequency. Only effective in Composition mode with a `status.` target. | | `FailOnEmpty` | bool | Optional. When true, the function will fail if the `users`, `groups`, or `servicePrincipals` lists are empty, or if their respective reference fields are empty lists. | | `identity.type` | string | Optional. Type of identity credentials to use. Valid values: `AzureServicePrincipalCredentials`, `AzureWorkloadIdentityCredentials`. Default is `AzureServicePrincipalCredentials` | @@ -296,6 +297,59 @@ target: "context.results" target: "context.[apiextensions.crossplane.io/environment].results" ``` +## Throttling Mitigation + +Microsoft Graph enforces API request throttling. The function offers two +complementary controls to reduce how often it calls the Graph API: + +- `skipQueryWhenTargetHasData` — skips the query whenever the target already + holds data. Useful when the data only needs to be fetched once. +- `queryInterval` — skips the query until a minimum interval has elapsed since + the last successful query, then refreshes the data. Useful when the data + should be kept reasonably fresh without querying on every reconcile. + +These are intended as alternative strategies. If both are set, the interval is +checked first, but `skipQueryWhenTargetHasData` takes over once the target holds +data, so the interval refresh never runs. Pick one per pipeline step; the +function emits a non-fatal warning when both are configured. + +### Query Interval + +`queryInterval` accepts a Go duration string (for example `10m`, `1h` or `90s`). +When set, the function records the timestamp of its last successful query +alongside the result stored at the status target and skips querying Microsoft +Graph again until the interval has elapsed — regardless of how frequently the +Composition reconciles. + +```yaml +target: "status.validatedUsers" +queryInterval: "10m" +``` + +The timestamp is persisted as an extra `lastQueryTime` element appended to the +result list at the status target, for example: + +```yaml +status: + validatedUsers: + - id: "a1b2c3" + displayName: "Jane Doe" + userPrincipalName: "jane@example.com" + mail: "jane@example.com" + - lastQueryTime: "2026-07-16T10:00:00Z" +``` + +Because the interval check reads the persisted timestamp back from the XR status +on subsequent reconciles, `queryInterval` is only effective in **Composition +mode with a `status.` target**. It has no effect for `context.` targets (context +is not persisted across reconciles) or in Operation mode, where the query +cadence is controlled by the `CronOperation` schedule or `WatchOperation` +instead. + +> Note: consumers of the result list should ignore the trailing element that +> carries `lastQueryTime` (it has no `id`), as it is metadata rather than a query +> result. + ## Using Reference Fields You can reference values from XR spec, status, or context instead of hardcoding them: diff --git a/example/README.md b/example/README.md index 91536ab..37540f3 100644 --- a/example/README.md +++ b/example/README.md @@ -121,3 +121,23 @@ crossplane render xr.yaml service-principal-example-context-ref.yaml functions.y ```shell crossplane render xr.yaml service-principal-example-spec-ref.yaml functions.yaml --function-credentials=./secrets/azure-creds.yaml -rc ``` + +### 5. Query Interval (Throttling) + +Throttle calls to Microsoft Graph with `queryInterval` (a Go duration string, e.g. `10m`). On a successful query the function appends a `lastQueryTime` timestamp to the results at the status target and, on later reconciles, skips querying until the interval has elapsed. It is only effective in Composition mode with a `status.` target. + +Run the query and observe the appended `lastQueryTime` element under `status.validatedUsers`: + +```shell +crossplane render xr.yaml user-validation-example-query-interval.yaml functions.yaml --function-credentials=./secrets/azure-creds.yaml -r +``` + +To observe the skip, render against an XR that already holds a recent `lastQueryTime`. `crossplane render` treats the XR file as the observed composite, so the function reads that timestamp and, while the interval has not elapsed, skips the query and emits a `FunctionSkip`/`IntervalLimit` condition instead of calling Graph: + +```shell +crossplane render xr-with-last-query-time.yaml user-validation-example-query-interval.yaml functions.yaml --function-credentials=./secrets/azure-creds.yaml -r +``` + +> `xr-with-last-query-time.yaml` uses a far-future `lastQueryTime` so the skip is deterministic; set it to a real recent time to test the natural elapsed boundary. The credentials are not used on the skip path (no Graph call is made), so they need not be valid for this command. + +> **macOS note:** these commands use `-r` (function results) rather than `-rc`. The `-c`/`--include-context` flag makes `crossplane render` v2.x run an internal context-extraction step over a unix socket bind-mounted into its Docker helper container, which Docker Desktop for macOS does not support (`connect: operation not supported`) — the render then hangs with no output. Since the query-interval results are written to a `status.` target, `-c` is unnecessary here. This is a known CLI bug ([crossplane/cli#161](https://github.com/crossplane/cli/issues/161)), fixed by [#163](https://github.com/crossplane/cli/pull/163) (context function now listens on TCP) but not yet in a tagged release as of CLI v2.4.0. Until then, drop `-c` on macOS, or run `crossplane render` on Linux / a CLI built from `main` if you need context output. diff --git a/example/e2e/README.md b/example/e2e/README.md new file mode 100644 index 0000000..b0dbef8 --- /dev/null +++ b/example/e2e/README.md @@ -0,0 +1,59 @@ +# queryInterval e2e (Composition mode) + +Manifests to exercise the `queryInterval` throttling feature end-to-end on a real +cluster, where the time-based skip/refresh loop can actually be observed across +reconciles (something `crossplane render` cannot show). + +## Files + +| File | Purpose | +|------|---------| +| `function.yaml` | Installs `function-msgraph` (pin the tag you want to test) | +| `composition.yaml` | UserValidation → `status.validatedUsers`, `queryInterval: "2m"` | +| `xr.yaml` | A composite resource instance | + +Reuses `../definition.yaml` (XRD) and the `azure-account-creds` secret built from +`../secrets/azure-creds.yaml` — see [Update Credentials](../README.md#update-credentials). +Populate it locally; keep real values out of commits. + +## Run + +```shell +kind create cluster --name msgraph-e2e +helm install crossplane crossplane-stable/crossplane \ + -n crossplane-system --create-namespace --version 2.3.3 --wait + +kubectl apply -f example/e2e/function.yaml +kubectl wait function.pkg.crossplane.io/function-msgraph --for=condition=Healthy --timeout=180s + +kubectl apply -f example/definition.yaml +kubectl apply -f example/e2e/composition.yaml +kubectl apply -f example/secrets/azure-creds.yaml # populate locally; keep real values out of commits +kubectl apply -f example/e2e/xr.yaml +``` + +## Verify + +```shell +# status carries the results plus a lastQueryTime element +kubectl get xr msgraph-query-interval-e2e -o jsonpath='{.status.validatedUsers}' | jq + +# within the interval, reconciles skip (condition is set on the XR) +kubectl get xr msgraph-query-interval-e2e -o jsonpath='{.status.conditions}' | jq # FunctionSkip/IntervalLimit + +# count real Graph calls vs skips in the function logs +POD=$(kubectl get pods -n crossplane-system -o name | grep msgraph) +kubectl logs -n crossplane-system "$POD" | grep -c 'Query Type' # real queries +kubectl logs -n crossplane-system "$POD" | grep -c 'interval limit' # skips +``` + +Force reconciles with `kubectl annotate xr msgraph-query-interval-e2e poke=$(date +%s) --overwrite`. +Poking repeatedly inside the 2m window leaves the query count flat; after 2m +elapses the next reconcile re-queries and `lastQueryTime` advances. + +## Notes + +- `queryInterval` is effective only in Composition mode with a `status.` target. +- The example XRD resolves to `LegacyCluster` scope under Crossplane v2, so the XR + is cluster-scoped (no namespace). +- Teardown: `kind delete cluster --name msgraph-e2e`. diff --git a/example/e2e/composition.yaml b/example/e2e/composition.yaml new file mode 100644 index 0000000..8b7bb54 --- /dev/null +++ b/example/e2e/composition.yaml @@ -0,0 +1,28 @@ +apiVersion: apiextensions.crossplane.io/v1 +kind: Composition +metadata: + name: user-validation-query-interval-e2e +spec: + compositeTypeRef: + apiVersion: example.crossplane.io/v1 + kind: XR + mode: Pipeline + pipeline: + - step: validate-user + functionRef: + name: function-msgraph + input: + apiVersion: msgraph.fn.crossplane.io/v1alpha1 + kind: Input + queryType: UserValidation + users: + - "yury@upbound.io" + target: "status.validatedUsers" + # Short interval so the skip/refresh loop is observable within a test session. + queryInterval: "2m" + credentials: + - name: azure-creds + source: Secret + secretRef: + namespace: crossplane-system + name: azure-account-creds diff --git a/example/e2e/function.yaml b/example/e2e/function.yaml new file mode 100644 index 0000000..0f7287b --- /dev/null +++ b/example/e2e/function.yaml @@ -0,0 +1,7 @@ +--- +apiVersion: pkg.crossplane.io/v1 +kind: Function +metadata: + name: function-msgraph +spec: + package: xpkg.upbound.io/upbound/function-msgraph:v0.7.0-rc3 diff --git a/example/e2e/xr.yaml b/example/e2e/xr.yaml new file mode 100644 index 0000000..df1c284 --- /dev/null +++ b/example/e2e/xr.yaml @@ -0,0 +1,8 @@ +apiVersion: example.crossplane.io/v1 +kind: XR +metadata: + name: msgraph-query-interval-e2e +spec: + userAccess: + emails: + - "yury@upbound.io" diff --git a/example/user-validation-example-query-interval.yaml b/example/user-validation-example-query-interval.yaml new file mode 100644 index 0000000..3ae75b6 --- /dev/null +++ b/example/user-validation-example-query-interval.yaml @@ -0,0 +1,38 @@ +apiVersion: apiextensions.crossplane.io/v1 +kind: Composition +metadata: + name: user-validation-example-query-interval +# Important: This function example requires an Azure AD app registration with Microsoft Graph API permissions: +# - User.Read.All +# - Directory.Read.All +spec: + compositeTypeRef: + apiVersion: example.crossplane.io/v1 + kind: XR + mode: Pipeline + pipeline: + - step: validate-user + functionRef: + name: function-msgraph + input: + apiVersion: msgraph.fn.crossplane.io/v1alpha1 + kind: Input + queryType: UserValidation + # Replace these with actual users in your directory + users: + - "admin@example.onmicrosoft.com" + - "user@example.onmicrosoft.com" + - "yury@upbound.io" + target: "status.validatedUsers" + # queryInterval throttles calls to Microsoft Graph independently of how + # often the Composition reconciles. The function records the timestamp of + # its last successful query alongside the result at the status target and + # skips querying again until the interval (a Go duration string) elapses. + # Only effective in Composition mode with a "status." target. + queryInterval: "10m" + credentials: + - name: azure-creds + source: Secret + secretRef: + namespace: crossplane-system + name: azure-account-creds diff --git a/example/xr-with-last-query-time.yaml b/example/xr-with-last-query-time.yaml new file mode 100644 index 0000000..6221379 --- /dev/null +++ b/example/xr-with-last-query-time.yaml @@ -0,0 +1,19 @@ +# XR that already carries a lastQueryTime in status.validatedUsers, used to +# demonstrate queryInterval skipping. `crossplane render` treats this XR as the +# observed composite, so the function reads the timestamp below and skips the +# Microsoft Graph query while the interval has not elapsed. +# +# The lastQueryTime here is set far in the future so the skip is deterministic +# regardless of when you run the command. Change it to a real recent time +# (within your queryInterval window) to exercise the natural elapsed boundary. +apiVersion: example.crossplane.io/v1 +kind: XR +metadata: + name: example-xr +status: + validatedUsers: + - id: "existing-user-id" + displayName: "Existing User" + userPrincipalName: "user@example.onmicrosoft.com" + mail: "user@example.onmicrosoft.com" + - lastQueryTime: "2099-01-01T00:00:00Z" diff --git a/fn.go b/fn.go index bee9863..4ac17d0 100644 --- a/fn.go +++ b/fn.go @@ -1019,11 +1019,15 @@ func (f *Function) putQueryResultToStatus(req *fnv1.RunFunctionRequest, rsp *fnv return err } + // Embed the query timestamp when interval limiting is configured so it can + // be read back on subsequent reconciles. + resultData := f.appendLastQueryTime(in, results) + // Update the specific status field statusField := strings.TrimPrefix(in.Target, "status.") - err = SetNestedKey(xrStatus, statusField, results) + err = SetNestedKey(xrStatus, statusField, resultData) if err != nil { - return errors.Wrapf(err, "cannot set status field %s to %v", statusField, results) + return errors.Wrapf(err, "cannot set status field %s to %v", statusField, resultData) } // Write the updated status field back into the composite resource @@ -1038,6 +1042,34 @@ func (f *Function) putQueryResultToStatus(req *fnv1.RunFunctionRequest, rsp *fnv return nil } +// appendLastQueryTime embeds the current timestamp into the query result so the +// interval-based skip logic can read it back on subsequent reconciles. It is a +// no-op unless queryInterval is configured. Array results (the intended +// structure) get an extra {"lastQueryTime": ...} element; map results get a +// lastQueryTime field. +func (f *Function) appendLastQueryTime(in *v1beta1.Input, results interface{}) interface{} { + if _, ok, _ := parseQueryInterval(in); !ok { + return results + } + + timestamp := f.timer.now() + switch data := results.(type) { + case []interface{}: + f.log.Debug("Added lastQueryTime element to array result", "target", in.Target, "queryInterval", *in.QueryInterval) + return append(data, map[string]interface{}{"lastQueryTime": timestamp}) + case map[string]interface{}: + data["lastQueryTime"] = timestamp + f.log.Debug("Added lastQueryTime to map result", "target", in.Target, "queryInterval", *in.QueryInterval) + return data + default: + f.log.Debug("Result data is neither array nor map, cannot add lastQueryTime", + "target", in.Target, + "resultType", fmt.Sprintf("%T", results), + "queryInterval", *in.QueryInterval) + return results + } +} + func putQueryResultToContext(req *fnv1.RunFunctionRequest, rsp *fnv1.RunFunctionResponse, in *v1beta1.Input, results interface{}, f *Function) error { contextField := strings.TrimPrefix(in.Target, "context.") data, err := structpb.NewValue(results) @@ -1171,6 +1203,22 @@ func (f *Function) validateAndPrepareInput(_ context.Context, req *fnv1.RunFunct return false } + // Validate queryInterval early so a misconfigured duration is surfaced + // clearly instead of silently disabling interval limiting. + _, intervalSet, err := parseQueryInterval(in) + if err != nil { + response.Fatal(rsp, err) + return false + } + + // Warn (non-fatal) when both throttling strategies are combined: + // skipQueryWhenTargetHasData takes precedence once the target has data, so + // it silently suppresses the queryInterval refresh. + if intervalSet && ptr.Deref(in.SkipQueryWhenTargetHasData, false) { + response.Warning(rsp, errors.New("both queryInterval and skipQueryWhenTargetHasData are set; skipQueryWhenTargetHasData takes precedence once the target has data, so the queryInterval refresh will not run")). + TargetCompositeAndClaim() + } + // Check if we should skip the query if f.shouldSkipQuery(req, in, rsp, inOperation) { // Set success condition @@ -1290,6 +1338,13 @@ func (f *Function) isValidTarget(target string) bool { // shouldSkipQuery checks if the query should be skipped. func (f *Function) shouldSkipQuery(req *fnv1.RunFunctionRequest, in *v1beta1.Input, rsp *fnv1.RunFunctionResponse, inOperation bool) bool { + // Interval-based skipping is evaluated first: if the configured interval has + // not yet elapsed since the last successful query, skip regardless of the + // other conditions. + if f.shouldSkipQueryDueToInterval(req, in, rsp, inOperation) { + return true + } + // Determine if we should skip the query when target has data var shouldSkipQueryWhenTargetHasData = false // Default to false to ensure continuous reconciliation if in.SkipQueryWhenTargetHasData != nil { @@ -1320,6 +1375,174 @@ func (f *Function) shouldSkipQuery(req *fnv1.RunFunctionRequest, in *v1beta1.Inp return false } +// parseQueryInterval parses the optional queryInterval duration string. +// It returns (0, false, nil) when the field is unset, (duration, true, nil) +// when it is set to a valid positive Go duration, and an error when it is set +// but cannot be parsed or is not positive. +func parseQueryInterval(in *v1beta1.Input) (time.Duration, bool, error) { + if in.QueryInterval == nil || *in.QueryInterval == "" { + return 0, false, nil + } + + d, err := time.ParseDuration(*in.QueryInterval) + if err != nil { + return 0, false, errors.Wrapf(err, "cannot parse queryInterval %q as a Go duration (e.g. \"10m\")", *in.QueryInterval) + } + if d <= 0 { + return 0, false, errors.Errorf("queryInterval must be a positive duration, got %q", *in.QueryInterval) + } + + return d, true, nil +} + +// shouldSkipQueryDueToInterval checks whether the configured queryInterval has +// not yet elapsed since the last successful query, in which case the query +// should be skipped. Interval limiting relies on reading the timestamp persisted +// in the status target on a previous reconcile, so it only applies to +// Composition mode with a "status." target. +func (f *Function) shouldSkipQueryDueToInterval(req *fnv1.RunFunctionRequest, in *v1beta1.Input, rsp *fnv1.RunFunctionResponse, inOperation bool) bool { + interval, ok, err := parseQueryInterval(in) + if err != nil || !ok { + // Invalid durations are surfaced during validation; treat as disabled here. + return false + } + + // The persisted timestamp is only readable back from the XR status in + // Composition mode, so interval limiting is unsupported elsewhere. + if inOperation || !strings.HasPrefix(in.Target, "status.") { + return false + } + + targetData, err := f.getStatusTargetData(req, in, inOperation) + if err != nil { + f.log.Debug("No existing target data for interval check, running query", "target", in.Target, "error", err) + return false + } + + lastQueryTime, err := f.extractLastQueryTime(targetData) + if err != nil { + f.log.Debug("No previous lastQueryTime found for interval check, running query", "target", in.Target, "error", err) + return false + } + + return f.checkIntervalLimit(lastQueryTime, interval, *in.QueryInterval, in.Target, rsp) +} + +// getStatusTargetData retrieves the current value stored at the status target +// from the observed XR status. +func (f *Function) getStatusTargetData(req *fnv1.RunFunctionRequest, in *v1beta1.Input, inOperation bool) (interface{}, error) { + xrStatus, _, err := f.getOXRAndStatus(req, inOperation) + if err != nil { + return nil, errors.Wrap(err, "cannot get XR status for interval check") + } + + statusField := strings.TrimPrefix(in.Target, "status.") + parts, err := ParseNestedKey(statusField) + if err != nil { + return nil, err + } + + currentValue := interface{}(xrStatus) + for _, k := range parts { + nestedMap, ok := currentValue.(map[string]interface{}) + if !ok { + return nil, errors.New("invalid nested structure") + } + nextValue, exists := nestedMap[k] + if !exists { + return nil, errors.New("no existing data") + } + currentValue = nextValue + } + + return currentValue, nil +} + +// extractLastQueryTime extracts and parses the lastQueryTime from target data, +// supporting both array results (the intended structure) and map results. +func (f *Function) extractLastQueryTime(targetData interface{}) (time.Time, error) { + if dataArray, ok := targetData.([]interface{}); ok { + return f.extractLastQueryTimeFromArray(dataArray) + } + if dataMap, ok := targetData.(map[string]interface{}); ok { + return f.extractLastQueryTimeFromMap(dataMap) + } + return time.Time{}, errors.New("target data is neither array nor map") +} + +// extractLastQueryTimeFromArray extracts lastQueryTime from the special element +// appended to array results. +func (f *Function) extractLastQueryTimeFromArray(dataArray []interface{}) (time.Time, error) { + for i := len(dataArray) - 1; i >= 0; i-- { + element, ok := dataArray[i].(map[string]interface{}) + if !ok { + continue + } + lastQueryTimeStr, exists := element["lastQueryTime"] + if !exists { + continue + } + lastQueryTimeString, ok := lastQueryTimeStr.(string) + if !ok { + continue + } + lastQueryTime, err := time.Parse(time.RFC3339, lastQueryTimeString) + if err != nil { + f.log.Debug("Cannot parse lastQueryTime from array element", "error", err) + return time.Time{}, err + } + return lastQueryTime, nil + } + return time.Time{}, errors.New("no lastQueryTime element found in array") +} + +// extractLastQueryTimeFromMap extracts lastQueryTime from a map result. +func (f *Function) extractLastQueryTimeFromMap(dataMap map[string]interface{}) (time.Time, error) { + lastQueryTimeStr, exists := dataMap["lastQueryTime"] + if !exists { + return time.Time{}, errors.New("no lastQueryTime field") + } + + lastQueryTimeString, ok := lastQueryTimeStr.(string) + if !ok { + return time.Time{}, errors.New("lastQueryTime is not a string") + } + + lastQueryTime, err := time.Parse(time.RFC3339, lastQueryTimeString) + if err != nil { + f.log.Debug("Cannot parse lastQueryTime", "error", err) + return time.Time{}, err + } + + return lastQueryTime, nil +} + +// checkIntervalLimit reports whether the interval has not yet elapsed since the +// last query, setting a FunctionSkip condition and returning true when it should +// be skipped. The current time is obtained from the timer to keep it testable. +func (f *Function) checkIntervalLimit(lastQueryTime time.Time, interval time.Duration, intervalDisplay, target string, rsp *fnv1.RunFunctionResponse) bool { + now, err := time.Parse(time.RFC3339, f.timer.now()) + if err != nil { + f.log.Debug("Cannot parse current time for interval check, running query", "error", err) + return false + } + + elapsed := now.Sub(lastQueryTime) + if elapsed < interval { + f.log.Info("Skipping query due to interval limit", + "target", target, + "interval", intervalDisplay, + "elapsed", elapsed.String()) + + response.ConditionTrue(rsp, "FunctionSkip", "IntervalLimit"). + WithMessage(fmt.Sprintf("Query skipped due to interval limit (%s)", intervalDisplay)). + TargetCompositeAndClaim() + return true + } + + return false +} + func (f *Function) queryDriftDetected(req *fnv1.RunFunctionRequest, inOperation bool) bool { _, oxr, err := f.getOXRAndStatus(req, inOperation) if err != nil { diff --git a/fn_test.go b/fn_test.go index 9c1c91d..15942d4 100644 --- a/fn_test.go +++ b/fn_test.go @@ -2967,6 +2967,377 @@ func TestRunFunction(t *testing.T) { }, }, }, + "SkipQueryDueToInterval": { + reason: "The Function should skip the query when the queryInterval has not elapsed since the last query", + args: args{ + ctx: context.Background(), + req: &fnv1.RunFunctionRequest{ + Meta: &fnv1.RequestMeta{Tag: "hello"}, + Input: resource.MustStructJSON(`{ + "apiVersion": "msgraph.fn.crossplane.io/v1alpha1", + "kind": "Input", + "queryType": "UserValidation", + "users": ["user@example.com"], + "target": "status.validatedUsers", + "queryInterval": "10m" + }`), + Observed: &fnv1.State{ + Composite: &fnv1.Resource{ + Resource: resource.MustStructJSON(`{ + "apiVersion": "example.org/v1", + "kind": "XR", + "status": { + "validatedUsers": [ + { + "id": "existing-user-id", + "displayName": "Existing User", + "userPrincipalName": "existing@example.com", + "mail": "existing@example.com" + }, + { + "lastQueryTime": "2024-12-31T23:55:00+01:00" + } + ] + } + }`), + }, + }, + Credentials: map[string]*fnv1.Credentials{ + "azure-creds": { + Source: &fnv1.Credentials_CredentialData{CredentialData: creds}, + }, + }, + }, + }, + want: want{ + rsp: &fnv1.RunFunctionResponse{ + Meta: &fnv1.ResponseMeta{Tag: "hello", Ttl: durationpb.New(response.DefaultTTL)}, + Conditions: []*fnv1.Condition{ + { + Type: "FunctionSkip", + Message: ptr.To("Query skipped due to interval limit (10m)"), + Status: fnv1.Status_STATUS_CONDITION_TRUE, + Reason: "IntervalLimit", + Target: fnv1.Target_TARGET_COMPOSITE_AND_CLAIM.Enum(), + }, + { + Type: "FunctionSuccess", + Status: fnv1.Status_STATUS_CONDITION_TRUE, + Reason: "Success", + Target: fnv1.Target_TARGET_COMPOSITE_AND_CLAIM.Enum(), + }, + }, + Desired: &fnv1.State{ + Composite: &fnv1.Resource{ + Resource: resource.MustStructJSON(`{ + "apiVersion": "example.org/v1", + "kind": "XR", + "status": { + "validatedUsers": [ + { + "id": "existing-user-id", + "displayName": "Existing User", + "userPrincipalName": "existing@example.com", + "mail": "existing@example.com" + }, + { + "lastQueryTime": "2024-12-31T23:55:00+01:00" + } + ] + }}`), + }, + }, + }, + }, + }, + "RunQueryWhenIntervalElapsed": { + reason: "The Function should run the query and refresh the timestamp when the queryInterval has elapsed", + args: args{ + ctx: context.Background(), + req: &fnv1.RunFunctionRequest{ + Meta: &fnv1.RequestMeta{Tag: "hello"}, + Input: resource.MustStructJSON(`{ + "apiVersion": "msgraph.fn.crossplane.io/v1alpha1", + "kind": "Input", + "queryType": "UserValidation", + "users": ["user@example.com"], + "target": "status.validatedUsers", + "queryInterval": "10m" + }`), + Observed: &fnv1.State{ + Composite: &fnv1.Resource{ + Resource: resource.MustStructJSON(`{ + "apiVersion": "example.org/v1", + "kind": "XR", + "status": { + "validatedUsers": [ + { + "id": "existing-user-id", + "displayName": "Existing User", + "userPrincipalName": "existing@example.com", + "mail": "existing@example.com" + }, + { + "lastQueryTime": "2024-12-31T22:00:00+01:00" + } + ] + } + }`), + }, + }, + Credentials: map[string]*fnv1.Credentials{ + "azure-creds": { + Source: &fnv1.Credentials_CredentialData{CredentialData: creds}, + }, + }, + }, + }, + want: want{ + rsp: &fnv1.RunFunctionResponse{ + Meta: &fnv1.ResponseMeta{Tag: "hello", Ttl: durationpb.New(response.DefaultTTL)}, + Conditions: []*fnv1.Condition{ + { + Type: "FunctionSuccess", + Status: fnv1.Status_STATUS_CONDITION_TRUE, + Reason: "Success", + Target: fnv1.Target_TARGET_COMPOSITE_AND_CLAIM.Enum(), + }, + }, + Results: []*fnv1.Result{ + { + Severity: fnv1.Severity_SEVERITY_NORMAL, + Message: `QueryType: "UserValidation"`, + Target: fnv1.Target_TARGET_COMPOSITE.Enum(), + }, + }, + Desired: &fnv1.State{ + Composite: &fnv1.Resource{ + Resource: resource.MustStructJSON(`{ + "apiVersion": "example.org/v1", + "kind": "XR", + "status": { + "validatedUsers": [ + { + "id": "test-user-id", + "displayName": "Test User", + "userPrincipalName": "user@example.com", + "mail": "user@example.com" + }, + { + "lastQueryTime": "2025-01-01T00:00:00+01:00" + } + ] + }}`), + }, + }, + }, + }, + }, + "RunQueryFirstTimeWithInterval": { + reason: "The Function should run the query and embed a lastQueryTime element on the first run when queryInterval is set and no prior data exists", + args: args{ + ctx: context.Background(), + req: &fnv1.RunFunctionRequest{ + Meta: &fnv1.RequestMeta{Tag: "hello"}, + Input: resource.MustStructJSON(`{ + "apiVersion": "msgraph.fn.crossplane.io/v1alpha1", + "kind": "Input", + "queryType": "UserValidation", + "users": ["user@example.com"], + "target": "status.validatedUsers", + "queryInterval": "10m" + }`), + Observed: &fnv1.State{ + Composite: &fnv1.Resource{ + Resource: resource.MustStructJSON(xr), + }, + }, + Credentials: map[string]*fnv1.Credentials{ + "azure-creds": { + Source: &fnv1.Credentials_CredentialData{CredentialData: creds}, + }, + }, + }, + }, + want: want{ + rsp: &fnv1.RunFunctionResponse{ + Meta: &fnv1.ResponseMeta{Tag: "hello", Ttl: durationpb.New(response.DefaultTTL)}, + Conditions: []*fnv1.Condition{ + { + Type: "FunctionSuccess", + Status: fnv1.Status_STATUS_CONDITION_TRUE, + Reason: "Success", + Target: fnv1.Target_TARGET_COMPOSITE_AND_CLAIM.Enum(), + }, + }, + Results: []*fnv1.Result{ + { + Severity: fnv1.Severity_SEVERITY_NORMAL, + Message: `QueryType: "UserValidation"`, + Target: fnv1.Target_TARGET_COMPOSITE.Enum(), + }, + }, + Desired: &fnv1.State{ + Composite: &fnv1.Resource{ + Resource: resource.MustStructJSON(`{ + "apiVersion": "example.org/v1", + "kind": "XR", + "metadata": { + "name": "cool-xr" + }, + "spec": { + "count": 2 + }, + "status": { + "validatedUsers": [ + { + "id": "test-user-id", + "displayName": "Test User", + "userPrincipalName": "user@example.com", + "mail": "user@example.com" + }, + { + "lastQueryTime": "2025-01-01T00:00:00+01:00" + } + ] + }}`), + }, + }, + }, + }, + }, + "InvalidQueryInterval": { + reason: "The Function should return a fatal result when queryInterval cannot be parsed as a Go duration", + args: args{ + ctx: context.Background(), + req: &fnv1.RunFunctionRequest{ + Meta: &fnv1.RequestMeta{Tag: "hello"}, + Input: resource.MustStructJSON(`{ + "apiVersion": "msgraph.fn.crossplane.io/v1alpha1", + "kind": "Input", + "queryType": "UserValidation", + "users": ["user@example.com"], + "target": "status.validatedUsers", + "queryInterval": "notaduration" + }`), + Observed: &fnv1.State{ + Composite: &fnv1.Resource{ + Resource: resource.MustStructJSON(xr), + }, + }, + Credentials: map[string]*fnv1.Credentials{ + "azure-creds": { + Source: &fnv1.Credentials_CredentialData{CredentialData: creds}, + }, + }, + }, + }, + want: want{ + rsp: &fnv1.RunFunctionResponse{ + Meta: &fnv1.ResponseMeta{Tag: "hello", Ttl: durationpb.New(response.DefaultTTL)}, + Results: []*fnv1.Result{ + { + Severity: fnv1.Severity_SEVERITY_FATAL, + Message: `cannot parse queryInterval "notaduration" as a Go duration (e.g. "10m"): time: invalid duration "notaduration"`, + Target: fnv1.Target_TARGET_COMPOSITE.Enum(), + }, + }, + Desired: &fnv1.State{ + Composite: &fnv1.Resource{ + Resource: resource.MustStructJSON(`{ + "apiVersion": "example.org/v1", + "kind": "XR", + "metadata": { + "name": "cool-xr" + }, + "spec": { + "count": 2 + } + }`), + }, + }, + }, + }, + }, + "WarnWhenIntervalAndSkipWhenTargetHasDataCombined": { + reason: "The Function should emit a non-fatal warning (and still run) when queryInterval and skipQueryWhenTargetHasData are both set", + args: args{ + ctx: context.Background(), + req: &fnv1.RunFunctionRequest{ + Meta: &fnv1.RequestMeta{Tag: "hello"}, + Input: resource.MustStructJSON(`{ + "apiVersion": "msgraph.fn.crossplane.io/v1alpha1", + "kind": "Input", + "queryType": "UserValidation", + "users": ["user@example.com"], + "target": "status.validatedUsers", + "queryInterval": "10m", + "skipQueryWhenTargetHasData": true + }`), + Observed: &fnv1.State{ + Composite: &fnv1.Resource{ + Resource: resource.MustStructJSON(xr), + }, + }, + Credentials: map[string]*fnv1.Credentials{ + "azure-creds": { + Source: &fnv1.Credentials_CredentialData{CredentialData: creds}, + }, + }, + }, + }, + want: want{ + rsp: &fnv1.RunFunctionResponse{ + Meta: &fnv1.ResponseMeta{Tag: "hello", Ttl: durationpb.New(response.DefaultTTL)}, + Conditions: []*fnv1.Condition{ + { + Type: "FunctionSuccess", + Status: fnv1.Status_STATUS_CONDITION_TRUE, + Reason: "Success", + Target: fnv1.Target_TARGET_COMPOSITE_AND_CLAIM.Enum(), + }, + }, + Results: []*fnv1.Result{ + { + Severity: fnv1.Severity_SEVERITY_WARNING, + Message: "both queryInterval and skipQueryWhenTargetHasData are set; skipQueryWhenTargetHasData takes precedence once the target has data, so the queryInterval refresh will not run", + Target: fnv1.Target_TARGET_COMPOSITE_AND_CLAIM.Enum(), + }, + { + Severity: fnv1.Severity_SEVERITY_NORMAL, + Message: `QueryType: "UserValidation"`, + Target: fnv1.Target_TARGET_COMPOSITE.Enum(), + }, + }, + Desired: &fnv1.State{ + Composite: &fnv1.Resource{ + Resource: resource.MustStructJSON(`{ + "apiVersion": "example.org/v1", + "kind": "XR", + "metadata": { + "name": "cool-xr" + }, + "spec": { + "count": 2 + }, + "status": { + "validatedUsers": [ + { + "id": "test-user-id", + "displayName": "Test User", + "userPrincipalName": "user@example.com", + "mail": "user@example.com" + }, + { + "lastQueryTime": "2025-01-01T00:00:00+01:00" + } + ] + }}`), + }, + }, + }, + }, + }, "QueryToContextField": { reason: "The Function should store results in context field", args: args{ diff --git a/input/v1beta1/input.go b/input/v1beta1/input.go index aa659ed..e290b13 100644 --- a/input/v1beta1/input.go +++ b/input/v1beta1/input.go @@ -67,6 +67,19 @@ type Input struct { // +optional SkipQueryWhenTargetHasData *bool `json:"skipQueryWhenTargetHasData,omitempty"` + // QueryInterval is an optional minimum interval between queries, expressed as + // a Go duration string (for example "10m", "1h" or "90s"). When set, the + // function records the timestamp of its last successful query alongside the + // result stored at the status target and skips querying Microsoft Graph again + // until the interval has elapsed, regardless of how frequently the Composition + // reconciles. This helps avoid Microsoft Graph API throttling. + // + // Only effective in Composition mode with a "status." target, because it + // relies on reading the persisted timestamp back from the XR status on + // subsequent reconciles. Leave empty to disable interval limiting (the default). + // +optional + QueryInterval *string `json:"queryInterval,omitempty"` + // FailOnEmpty controls whether the function should fail when input lists are empty. // If true, the function will error on empty input lists. // If false or unset, empty lists are valid and will result in an empty list at the target. diff --git a/input/v1beta1/zz_generated.deepcopy.go b/input/v1beta1/zz_generated.deepcopy.go index 7d40fb2..0031c7b 100644 --- a/input/v1beta1/zz_generated.deepcopy.go +++ b/input/v1beta1/zz_generated.deepcopy.go @@ -91,6 +91,16 @@ func (in *Input) DeepCopyInto(out *Input) { *out = new(bool) **out = **in } + if in.QueryInterval != nil { + in, out := &in.QueryInterval, &out.QueryInterval + *out = new(string) + **out = **in + } + if in.FailOnEmpty != nil { + in, out := &in.FailOnEmpty, &out.FailOnEmpty + *out = new(bool) + **out = **in + } if in.Identity != nil { in, out := &in.Identity, &out.Identity *out = new(Identity) diff --git a/package/input/msgraph.fn.crossplane.io_inputs.yaml b/package/input/msgraph.fn.crossplane.io_inputs.yaml index 894f1c7..c1c25f5 100644 --- a/package/input/msgraph.fn.crossplane.io_inputs.yaml +++ b/package/input/msgraph.fn.crossplane.io_inputs.yaml @@ -3,7 +3,7 @@ apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: - controller-gen.kubebuilder.io/version: v0.19.0 + controller-gen.kubebuilder.io/version: v0.20.1 name: inputs.msgraph.fn.crossplane.io spec: group: msgraph.fn.crossplane.io @@ -28,6 +28,12 @@ spec: may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources type: string + failOnEmpty: + description: |- + FailOnEmpty controls whether the function should fail when input lists are empty. + If true, the function will error on empty input lists. + If false or unset, empty lists are valid and will result in an empty list at the target. + type: boolean group: description: Group is a single group name for group membership queries type: string @@ -67,6 +73,19 @@ spec: type: string metadata: type: object + queryInterval: + description: |- + QueryInterval is an optional minimum interval between queries, expressed as + a Go duration string (for example "10m", "1h" or "90s"). When set, the + function records the timestamp of its last successful query alongside the + result stored at the status target and skips querying Microsoft Graph again + until the interval has elapsed, regardless of how frequently the Composition + reconciles. This helps avoid Microsoft Graph API throttling. + + Only effective in Composition mode with a "status." target, because it + relies on reading the persisted timestamp back from the XR status on + subsequent reconciles. Leave empty to disable interval limiting (the default). + type: string queryType: description: |- QueryType defines the type of Microsoft Graph API query to perform